Lab book Fluotracify

• gitflow-avh (magit-flow) to follow the flow
• possibly https://github.com/magit/magit-annex for large files. Follow this: https://git-annex.branchable.com/walkthrough/
• maybe check out git-toolbelt at some point https://github.com/nvie/git-toolbelt#readme with https://nvie.com/posts/git-power-tools/

• emacs jupyter for running and connecting to kernel on server: https://github.com/dzop/emacs-jupyter
• if I actually still would use .ipynb files, these might come handy:
  • jupytext: https://github.com/mwouts/jupytext
  • nbstripout: https://github.com/kynan/nbstripout

• https://docs.faculty.ai/user-guide/experiments/index.html and https://docs.microsoft.com/en-us/azure/databricks/_static/notebooks/hls-image-processing/02-image-segmentation-dl.html

• https://www.tensorflow.org/tensorboard/image_summaries

Different applications can be run on the remote compute node. If I want to access them at the local machine, and open them with the browser, I use this tunneling script.

ssh -t -t ara -L $port:localhost:$port ssh $node -L $port:Localhost:$port

Apps I use that way:

• Jupyter lab for running Python 3-Kernels
• TensorBoard
• Mlflow ui

Starting a jupyter instance on a server where the necessary libraries are installed is easy using this script:

conda activate tf
export PORT=9999
export XDG_RUNTIME_DIR=''
export XDG_RUNTIME_DIR=""
jupyter lab --no-browser --port=$PORT

On the compute node of the HPC, the users' environment is managed through module files using the system Lmod. The export XDG_RUNTIME_DIR statements are needed because of a jupyter bug which did not let it start. Right now, ob-tmux does not support a :var header like normal org-babel does. So the $port variable has to be set here in the template.

Now this port has to be tunnelled on our local computer (See SSH tunneling). While the tmux session above keeps running, no matter if Emacs is running or not, this following ssh tunnel needs to be active locally to connect to the notebook. If you close Emacs, it would need to be reestablished

• prerequisite: tmux versions need to be the same locally and on the server. Let's verify that now.
  • the local tmux version:

    tmux -V
    

    tmux 3.0a
    

  • the remote tmux version:

    ssh ara tmux -V
    

    ye53nis@ara-login01.rz.uni-jena.de's	password:
    tmux	3.0a

• as is described in the ob-tmux readme, the following code snippet creates a socket on the remote machine and forwards this socket to the local machine (note that socket_path was introduced in tmux version 2.2)

  REMOTE_SOCKET=$(ssh ara 'tmux ls -F "#{socket_path}"' | head -1)
  echo $REMOTE_SOCKET
  ssh ara -tfN \
      -L ~/.tmux-local-socket-remote-machine:$REMOTE_SOCKET
  

  ye53nis@ara-login01.rz.uni-jena.de's	password:
  /tmp/tmux-67339/default
  >	ye53nis@ara-login01.rz.uni-jena.de's	password:

• now a new tmux session with name ob-NAME is created when using a code block which looks like this: #+BEGIN_SRC tmux :socket ~/.tmux-local-socket-remote-machine :session NAME
• Commands can be sent now to the remote tmux session, BUT note that the output is not printed yet
• there is a workaround for getting output back to our LabBook.org: A script which allows to print the output from the tmux session in an #+begin_example-Block below the tmux block by pressing C-c C-o or C-c C-v C-o when the pointer is inside the tmux block.

Emacs-jupyter aims to be an API for a lot of functionalities of the jupyter project. The documentation can be found on GitHub.

1. For the whole document: connect to a running jupyter instance
   1. M-x jupyter-server-list-kernels
      1. set server URL, e.g. http://localhost:8889
      2. set websocket URL, e.g. http://localhost:8889
   2. two possibilities
      1. kernel already exists \(\to\) list of kernels and kernel-ID is displayed
      2. kernel does not exist \(\to\) prompt asks if you want to start one \(\to\) yes \(\to\) type kernel you want to start, e.g. Python 3
2. In the subtree where you want to use jupyter-python blocks with org babel
   1. set the :header-args:jupyter-python :session /jpy:localhost#kernel:8889-ID
   2. customize the output folder using the following org-mode variable:

      (setq org-babel-jupyter-resource-directory "./data/exp-test/plots")
      

      ./data/exp-test/plots
      

3. For each individual block, the following customizations might be useful
   1. jupyter kernels can return multiple kinds of rich output (images, html, …) or scalar data (plain text, numbers, lists, …). To force a plain output, use :results scalar. To show the output in the minibuffer only, use :results silent
   2. to change the priority of different rich outputs, use :display header argument, e.g. :display text/plain text/html prioritizes plain text over html. All supported mimetypes in default order:
      1. text/org
      2. image/svg+xml, image/jpeg, image/png
      3. text/html
      4. text/markdown
      5. text/latex
      6. text/plain
   3. We can set jupyter to output pandas DataFrames as org tables automatically using the source block header argument :pandoc t
   4. useful keybindings
      • M-i to open the documentation for wherever your pointer is (like pressing Shift-TAB in Jupyter notebooks)
      • C-c C-i to interrupt the kernel, C-c C-r to restart the kernel

• I am partial to the twitter bootstrap theme of html, since I like it's simple design, but clear structure with a nice table of contents at the side → the following org mode extension supports a seemless export to twitter bootstrap html: https://github.com/marsmining/ox-twbs
• when installed, the export can be triggered via the command (org-twbs-export-as-html) or via the keyboard shortcut for export C-c C-e followed by w for Twitter bootstrap and h for saving the .html
• Things to configure:
  • in general, there are multiple export options: https://orgmode.org/manual/Export-Settings.html
  • E.g. I set 2 #+OPTIONS keywords at the begin of the file: toc:4 and H:4 which make sure that in my export my sidebar table of contents will show numbered headings till a depth of 4.
  • I configured my code blocks so that they will not be evaluated when exporting (I would recommend this especially if you only export for archiving) and that both the code block and the output will be exported with the keyword: #+PROPERTY: header-args :eval never-export :exports both
  • To discriminate between code blocks for different languages I gave each of them a distinct colour using #+HTML_HEAD_EXTRA: <style... (see above)
  • I had to configure a style for table, so that the
    • display: block; overflow-x: auto; gets the table to be restricted to the width of the text and if it is larger, activates scrolling
    • white-space: nowrap; makes it that there is no wrap in a column, so it might be broader, but better readable if you have scrolling anyway
• Things to do before exporting / Troubleshooting while exporting:
  • when using a dark theme for you emacs, the export of the code blocks might show some ugly dark backgrounds from the theme. If this becomes an issue, change to a light theme for the export with M-x (load-theme) and choose solarized-light
  • only in the data branch you set the git tags after merging. If you want to show them here, execute the corresponding function in Git TAGs
  • make sure your file links work properly! I recommend referencing your files relatively (e.g. [ [ f ile:./data/exp-XXXXXX-test/test.png]] without spaces). Otherwise there will be errors in your Messages buffer
  • There might be errors with your code blocks
    • e.g. the export function expects you to assign a default variable to your functions
    • if you call a function via the #+CALL mechanism, it wants you to include two parentheses for the function, e.g. #+CALL: test()
  • check indentation of code blocks inside lists
  • add a details block around large output cells. This makes them expandable. I added some #+HTML_HEAD_EXTRA: <style... inspired by alhassy. That's how the details block looks like:

    #+begin_details
    
    #+end_details
    

  • If you reference a parameter with an underscore in the name, use the org markdown tricks to style them like code (== or ~~), otherwise the part after the underscore will be rendered like a subscript: under_score vs under_score
• Things to do after exporting:
  • In my workflow, the exported LabBook.html with the overview of all experiments is in the data folder. If you move the file, you will have to fix the file links for the new location, e.g. via "Find and replace" M-%:
    • if you move the org file → in the org file find [[file:./data/ and replace with [[file:./ → then export with C-c C-e w h
    • if you export first with C-c C-e w h and move the html file to data → in the html file find ./data and replace with .

git push origin --tags
git tag -n1

exp-200402-test Merge branch 'exp-200402-test' into data
exp-200520-unet Merge branch 'exp-310520-unet' into data
exp-200531-unet Merge branch 'heads/exp-310520-unet' into data
exp-201231-clustsim exp-201231-clustsim
exp-210204-unet Add exp-210204-unet LabBook part 3
exp-310520-unet move exp-310520-unet to data branch manually

• cloned from dwaithe with refactoring for Python 3-compatibility

• Add #+HTML_HEAD_EXTRA: <style... for table to enable scrolling if the table overflows

• Add details blocks, corresponding #+HTML_HEAD_EXTRA: <style... and documentation in Notes on archiving

• Rename master branch to main branch

• Add #+OPTIONS: H:4 and #+OPTIONS: toc:4 to show up to 4 levels of depth in the html (twbs) export of this LabBook in the table of contents at the side
• I added Notes on archiving

• update "jupyter scripts" in Template for data entry and setup notes: for new conda environment on server (now conda activate tf-nightly)

• extend general documentation in README
• Add code block examples
• extend documentation on experiment workflow
• move setup notes from README to "Template for data entry and setup notes"
• remove emacs-lisp code for custom tmux block functions (not relevant enough)
• change named "jpt-tmux" from starting a jupyter notebook to starting jupyter lab. Load a conda environment instead of using Lmod's module load

• extend documentation on git model
• extend documentation on jupyter setup

• added parts of README which describe the experimental process
• added templates for system metadata, tmux, jupyter setup
• added organization of code

• set up lab book and form git repo accoring to setup by Luka Stanisic et al

Test:

pwd

(tensorflow_nightly) [ye53nis@login01 drmed-git]$ pwd
/beegfs/ye53nis/drmed-git

pwd

srun -p s_standard --time=7-10:00:00 --ntasks-per-node=24 --mem-per-cpu=2000 --pty bash

(base) [ye53nis@node151 drmed-git]$

1. Request compute node via tmux

   srun -p s_standard --time=7-10:00:00 --ntasks-per-node=24 --mem-per-cpu=2000 --pty bash
   

   (base) [ye53nis@node189 drmed-git]$
   

   cd /home/ye53nis/DOKTOR
   

1. Start Jupyter Lab

   [I 10:54:50.672 LabApp] JupyterLab extension loaded from /home/ye53nis/.conda/envs/tensorflow_nightly/lib/python3.8/site-packages/jupyterlab
   [I 10:54:50.673 LabApp] JupyterLab application directory is /home/ye53nis/.conda/envs/tensorflow_nightly/share/jupyter/lab
   [I 10:54:50.678 LabApp] Serving notebooks from local directory: /home/ye53nis/DOKTOR
   [I 10:54:50.678 LabApp] The Jupyter Notebook is running at:
   [I 10:54:50.678 LabApp] http://localhost:8889/?token=b210da5d453ac75f8f246e3c23917c94578c516ecfe0d95b
   [I 10:54:50.678 LabApp]  or http://127.0.0.1:8889/?token=b210da5d453ac75f8f246e3c23917c94578c516ecfe0d95b
   [I 10:54:50.678 LabApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
   [C 10:54:50.724 LabApp]
   
       To access the notebook, open this file in a browser:
           file:///home/ye53nis/.local/share/jupyter/runtime/nbserver-365563-open.html
       Or copy and paste one of these URLs:
           http://localhost:8889/?token=b210da5d453ac75f8f246e3c23917c94578c516ecfe0d95b
        or http://127.0.0.1:8889/?token=b210da5d453ac75f8f246e3c23917c94578c516ecfe0d95b
   

I started a Python3 kernel using jupyter-server-list-kernels. Then I added the kernel ID to the :PROPERTIES: drawer of this (and following) subtrees.

python3           7fe3d66b-d802-4f0f-b105-9536a917b816   a few seconds ago    starting   0

Test:

No of CPUs in system: 72
No of CPUs the current process can use: 24
load average: (1.06, 1.01, 0.97)
os.uname():  posix.uname_result(sysname='Linux', nodename='node218', release='3.10.0-957.1.3.el7.x86_64', version='#1 SMP Thu Nov 29 14:49:43 UTC 2018', machine='x86_64')
PID of process: 10254
RAM total: 199G, RAM used: 6.0G, RAM free: 181G
the current directory: /home/ye53nis/DOKTOR
My disk usage:
Filesystem           Size  Used Avail Use% Mounted on
/dev/sda1             50G  3.2G   47G   7% /
devtmpfs              94G     0   94G   0% /dev
tmpfs                 94G  725M   94G   1% /dev/shm
tmpfs                 94G   75M   94G   1% /run
tmpfs                 94G     0   94G   0% /sys/fs/cgroup
nfs02-ib:/data01      88T   61T   27T  70% /data01
nfs03-ib:/pool/work  100T   79T   22T  79% /nfsdata
nfs01-ib:/home        80T   61T   20T  76% /home
nfs01-ib:/cluster    2.0T  316G  1.7T  16% /cluster
/dev/sda3            6.0G  412M  5.6G   7% /var
/dev/sda6            169G  2.8G  166G   2% /local
/dev/sda5            2.0G   34M  2.0G   2% /tmp
beegfs_nodev         524T  453T   72T  87% /beegfs
tmpfs                 19G     0   19G   0% /run/user/67339
/bin/sh: conda: command not found
{'SLURM_CHECKPOINT_IMAGE_DIR': '/var/slurm/checkpoint',
 'SLURM_NODELIST': 'node218',
 'SLURM_JOB_NAME': 'bash',
 'XDG_SESSION_ID': '8541',
 'SLURMD_NODENAME': 'node218',
 'SLURM_TOPOLOGY_ADDR': 'node218',
 'SLURM_NTASKS_PER_NODE': '24',
 'HOSTNAME': 'login01',
 'SLURM_PRIO_PROCESS': '0',
 'SLURM_SRUN_COMM_PORT': '45911',
 'SHELL': '/bin/bash',
 'TERM': 'xterm-color',
 'SLURM_JOB_QOS': 'qstand',
 'SLURM_PTY_WIN_ROW': '52',
 'HISTSIZE': '1000',
 'TMPDIR': '/tmp',
 'SLURM_TOPOLOGY_ADDR_PATTERN': 'node',
 'SSH_CLIENT': '10.231.190.186 48592 22',
 'CONDA_SHLVL': '2',
 'CONDA_PROMPT_MODIFIER': '(tensorflow_nightly) ',
 'GSETTINGS_SCHEMA_DIR_CONDA_BACKUP': '',
 'WINDOWID': '0',
 'OLDPWD': '/beegfs/ye53nis/drmed-git',
 'QTDIR': '/usr/lib64/qt-3.3',
 'QTINC': '/usr/lib64/qt-3.3/include',
 'SSH_TTY': '/dev/pts/57',
 'QT_GRAPHICSSYSTEM_CHECKED': '1',
 'SLURM_NNODES': '1',
 'USER': 'ye53nis',
 'http_proxy': 'http://internet4nzm.rz.uni-jena.de:3128',
 'LS_COLORS': 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:',
 'CONDA_EXE': '/cluster/miniconda3/bin/conda',
 'SLURM_STEP_NUM_NODES': '1',
 'SLURM_JOBID': '392142',
 'SRUN_DEBUG': '3',
 'SLURM_NTASKS': '24',
 'SLURM_LAUNCH_NODE_IPADDR': '192.168.192.5',
 'SLURM_STEP_ID': '0',
 'TMUX': '/tmp/tmux-67339/default,47311,2',
 '_CE_CONDA': '',
 'CONDA_PREFIX_1': '/cluster/miniconda3',
 'SLURM_STEP_LAUNCHER_PORT': '45911',
 'SLURM_TASKS_PER_NODE': '24',
 'MAIL': '/var/spool/mail/ye53nis',
 'PATH': '/home/ye53nis/.conda/envs/tensorflow_nightly/bin:/home/lex/Programme/miniconda3/envs/tensorflow_env/bin:/home/lex/Programme/miniconda3/condabin:/home/lex/.local/bin:/bin:/usr/bin:/usr/local/bin:/usr/local/sbin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/var/lib/snapd/snap/bin:/home/lex/Programme/miniconda3/bin:/usr/sbin:/home/ye53nis/.local/bin:/home/ye53nis/bin',
 'GSETTINGS_SCHEMA_DIR': '/home/ye53nis/.conda/envs/tensorflow_nightly/share/glib-2.0/schemas',
 'SLURM_WORKING_CLUSTER': 'hpc:192.168.192.1:6817:8448',
 'SLURM_JOB_ID': '392142',
 'CONDA_PREFIX': '/home/ye53nis/.conda/envs/tensorflow_nightly',
 'SLURM_JOB_USER': 'ye53nis',
 'SLURM_STEPID': '0',
 'PWD': '/home/ye53nis/DOKTOR',
 'SLURM_SRUN_COMM_HOST': '192.168.192.5',
 'LANG': 'en_US.UTF-8',
 'SLURM_PTY_WIN_COL': '206',
 'SLURM_UMASK': '0022',
 'MODULEPATH': '/usr/share/Modules/modulefiles:/etc/modulefiles:/cluster/modulefiles',
 'SLURM_JOB_UID': '67339',
 'LOADEDMODULES': '',
 'SLURM_NODEID': '0',
 'TMUX_PANE': '%2',
 'SLURM_SUBMIT_DIR': '/beegfs/ye53nis/drmed-git',
 'SLURM_TASK_PID': '696',
 'SLURM_NPROCS': '24',
 'SLURM_CPUS_ON_NODE': '24',
 'SLURM_DISTRIBUTION': 'block',
 'https_proxy': 'https://internet4nzm.rz.uni-jena.de:3128',
 'SLURM_PROCID': '0',
 'HISTCONTROL': 'ignoredups',
 '_CE_M': '',
 'SLURM_JOB_NODELIST': 'node218',
 'SLURM_PTY_PORT': '37811',
 'HOME': '/home/ye53nis',
 'SHLVL': '3',
 'SLURM_LOCALID': '0',
 'SLURM_JOB_GID': '13280',
 'SLURM_JOB_CPUS_PER_NODE': '24',
 'SLURM_CLUSTER_NAME': 'hpc',
 'SLURM_GTIDS': '0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23',
 'SLURM_SUBMIT_HOST': 'login01',
 'SLURM_JOB_PARTITION': 's_standard',
 'MATHEMATICA_HOME': '/cluster/apps/mathematica/11.3',
 'CONDA_PYTHON_EXE': '/cluster/miniconda3/bin/python',
 'LOGNAME': 'ye53nis',
 'SLURM_STEP_NUM_TASKS': '24',
 'QTLIB': '/usr/lib64/qt-3.3/lib',
 'SLURM_JOB_ACCOUNT': 'iaob',
 'SLURM_JOB_NUM_NODES': '1',
 'MODULESHOME': '/usr/share/Modules',
 'CONDA_DEFAULT_ENV': 'tensorflow_nightly',
 'LESSOPEN': '||/usr/bin/lesspipe.sh %s',
 'SLURM_STEP_TASKS_PER_NODE': '24',
 'PORT': '8889',
 'SLURM_STEP_NODELIST': 'node218',
 'DISPLAY': ':0',
 'XDG_RUNTIME_DIR': '',
 'XAUTHORITY': '/tmp/xauth-1000-_0',
 'BASH_FUNC_module()': '() {  eval `/usr/bin/modulecmd bash $*`\n}',
 '_': '/home/ye53nis/.conda/envs/tensorflow_nightly/bin/jupyter',
 'KERNEL_LAUNCH_TIMEOUT': '40',
 'JPY_PARENT_PID': '7566',
 'CLICOLOR': '1',
 'PAGER': 'cat',
 'GIT_PAGER': 'cat',
 'MPLBACKEND': 'module://ipykernel.pylab.backend_inline'}

mlflow run . -P fluotracify_path=/beegfs/ye53nis/drmed-git/src/ -P epochs=2 -P learning_rate=None -P csv_path=/beegfs/ye53nis/saves/firstartifact_Mar2020 -P steps_per_epoch=400 -P validation_steps=200

(tensorflow_nightly) [ye53nis@node171 drmed-git]$ mlflow run . -P fluotracify_path=/beegfs/ye53nis/drmed-git/src/ -P epochs=2 -P learning_rate=None -P csv_path=/beegfs/ye53nis/saves/firstartifact_Mar2020 -P
   steps_per_epoch=400 -P validation_steps=200
  WARNING:root:Malformed experiment '1'. Detailed error Yaml file './data/mlruns/1/meta.yaml' does not exist.
  Traceback (most recent call last):
    File "/home/ye53nis/.conda/envs/tensorflow_nightly/lib/python3.8/site-packages/mlflow/store/tracking/file_store.py", line 197, in list_experiments
      experiment = self._get_experiment(exp_id, view_type)
    File "/home/ye53nis/.conda/envs/tensorflow_nightly/lib/python3.8/site-packages/mlflow/store/tracking/file_store.py", line 260, in _get_experiment
      meta = read_yaml(experiment_dir, FileStore.META_DATA_FILE_NAME)
    File "/home/ye53nis/.conda/envs/tensorflow_nightly/lib/python3.8/site-packages/mlflow/utils/file_utils.py", line 167, in read_yaml
      raise MissingConfigException("Yaml file '%s' does not exist." % file_path)
  mlflow.exceptions.MissingConfigException: Yaml file './data/mlruns/1/meta.yaml' does not exist.
  INFO: 'exp-310520-unet' does not exist. Creating a new experiment
  WARNING:root:Malformed experiment '1'. Detailed error Yaml file './data/mlruns/1/meta.yaml' does not exist.
  Traceback (most recent call last):
    File "/home/ye53nis/.conda/envs/tensorflow_nightly/lib/python3.8/site-packages/mlflow/store/tracking/file_store.py", line 197, in list_experiments
      experiment = self._get_experiment(exp_id, view_type)
    File "/home/ye53nis/.conda/envs/tensorflow_nightly/lib/python3.8/site-packages/mlflow/store/tracking/file_store.py", line 260, in _get_experiment
      meta = read_yaml(experiment_dir, FileStore.META_DATA_FILE_NAME)
    File "/home/ye53nis/.conda/envs/tensorflow_nightly/lib/python3.8/site-packages/mlflow/utils/file_utils.py", line 167, in read_yaml
      raise MissingConfigException("Yaml file '%s' does not exist." % file_path)
  mlflow.exceptions.MissingConfigException: Yaml file './data/mlruns/1/meta.yaml' does not exist.
  WARNING:root:Malformed experiment '1'. Detailed error Yaml file './data/mlruns/1/meta.yaml' does not exist.
  Traceback (most recent call last):
    File "/home/ye53nis/.conda/envs/tensorflow_nightly/lib/python3.8/site-packages/mlflow/store/tracking/file_store.py", line 197, in list_experiments
      experiment = self._get_experiment(exp_id, view_type)
    File "/home/ye53nis/.conda/envs/tensorflow_nightly/lib/python3.8/site-packages/mlflow/store/tracking/file_store.py", line 260, in _get_experiment
      meta = read_yaml(experiment_dir, FileStore.META_DATA_FILE_NAME)
    File "/home/ye53nis/.conda/envs/tensorflow_nightly/lib/python3.8/site-packages/mlflow/utils/file_utils.py", line 167, in read_yaml
      raise MissingConfigException("Yaml file '%s' does not exist." % file_path)
  mlflow.exceptions.MissingConfigException: Yaml file './data/mlruns/1/meta.yaml' does not exist.
  2020/05/31 14:00:34 INFO mlflow.projects: === Created directory /tmp/tmpfhabm3p4 for downloading remote URIs passed to arguments of type 'path' ===
  2020/05/31 14:00:34 INFO mlflow.projects: === Running command 'source /cluster/miniconda3/bin/../etc/profile.d/conda.sh && conda activate mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b 1>&2 && python src/f
  luotracify/training/train.py /beegfs/ye53nis/drmed-git/src 5 0.2 16384 None 2 /beegfs/ye53nis/saves/firstartifact_Mar2020 400 200' in run with ID 'b37371694a004638a6cd7a94d4d2e77f' ===
  2020-05-31 14:00:38.667018: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No
   such file or directory
  2020-05-31 14:00:38.667107: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
  /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/framework/load_library.py:23: DeprecationWarning: the imp module is deprecated in favo
  ur of importlib; see the module's documentation for alternative uses
    import imp
  2.3.0-dev20200527
  2020-05-31 14:00:42.422965: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file
   or directory
  2020-05-31 14:00:42.422998: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: UNKNOWN ERROR (303)
  2020-05-31 14:00:42.423020: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node171): /proc/driver/nvidia/version does not exist
  GPUs:  []
  train 0 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set027.csv
  train 1 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set087.csv
  train 2 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set003.csv
  train 3 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set056.csv
  train 4 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set076.csv
  train 5 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set094.csv
  train 6 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set017.csv
  train 7 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set074.csv
  train 8 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set055.csv
  train 9 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set096.csv
  train 10 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set054.csv
  train 11 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set093.csv
  train 12 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set079.csv
  train 13 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set014.csv
  train 14 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set008.csv
  train 15 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set031.csv
  train 16 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set023.csv
  train 17 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set025.csv
  train 18 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set034.csv
  train 19 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set009.csv
  train 20 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set044.csv
  train 21 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set063.csv
  train 22 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set004.csv
  train 23 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set072.csv
  train 24 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set046.csv
  train 25 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set049.csv
  train 26 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set007.csv
  train 27 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set100.csv
  train 28 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set083.csv
  train 29 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set077.csv
  train 30 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set061.csv
  train 31 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set081.csv
  train 32 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set091.csv
  train 33 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set069.csv
  train 34 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set052.csv
  train 35 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set028.csv
  train 36 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set019.csv
  train 37 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set057.csv
  train 38 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set064.csv
  train 39 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set075.csv
  train 40 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set002.csv
  train 41 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set062.csv
  train 42 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set043.csv
  train 43 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set042.csv
  train 44 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set005.csv
  train 45 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set016.csv
  train 46 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set018.csv
  train 47 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set041.csv
  train 48 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set039.csv
  train 49 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set006.csv
  train 50 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set092.csv
  train 51 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set060.csv
  train 52 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set001.csv
  train 53 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set035.csv
  train 54 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set029.csv
  train 55 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set051.csv
  train 56 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set012.csv
  train 57 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set036.csv
  train 58 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set024.csv
  train 59 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set053.csv
  train 60 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set011.csv
  train 61 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set032.csv
  train 62 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set067.csv
  train 63 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set058.csv
  train 64 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set080.csv
  train 65 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set086.csv
  train 66 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set033.csv
  train 67 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set085.csv
  train 68 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set015.csv
  train 69 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set090.csv
  train 70 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set020.csv
  train 71 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set030.csv
  train 72 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set050.csv
  train 73 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set098.csv
  train 74 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set099.csv
  train 75 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set070.csv
  train 76 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set021.csv
  train 77 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set095.csv
  train 78 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set073.csv
  train 79 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set078.csv
  test 80 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set026.csv
  test 81 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set038.csv
  test 82 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set082.csv
  test 83 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set047.csv
  test 84 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set040.csv
  test 85 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set066.csv
  test 86 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set059.csv
  test 87 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set013.csv
  test 88 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set089.csv
  test 89 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set071.csv
  test 90 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set088.csv
  test 91 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set037.csv
  test 92 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set022.csv
  test 93 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set084.csv
  test 94 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set010.csv
  test 95 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set097.csv
  test 96 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set068.csv
  test 97 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set065.csv
  test 98 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set048.csv
  test 99 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set045.csv
  shapes of feature dataframe: (16384, 8000) and label dataframe: (16384, 8000)
  shapes of feature dataframe: (16384, 2000) and label dataframe: (16384, 2000)

  for each 20,000 timestap trace there are the following numbers of corrupted timesteps:
   label001_1    16384
  label001_1    16384
  label001_1    16384
  label001_1    16384
  label001_1    16384
  dtype: int64
  2020-05-31 14:05:15.534482: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with Intel(R) MKL-DNN to use the following CPU instructions in performance-critical opera
  tions:  AVX2 AVX512F FMA
  To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
  2020-05-31 14:05:15.544530: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2300000000 Hz
  2020-05-31 14:05:15.545867: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x557b10ed44b0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
  2020-05-31 14:05:15.545896: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
  number of training examples: 6400, number of validation examples: 1600

  ------------------------
  number of test examples: 2000

  input - shape:   (None, 16384, 1)
  output - shape:  (None, 16384, 1)
  2020-05-31 14:05:22.334555: I tensorflow/core/profiler/lib/profiler_session.cc:163] Profiler session started.
  /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/training/tracking/data_structures.py:739: DeprecationWarning: Using or importing the A
  BCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working
    if not isinstance(wrapped_dict, collections.Mapping):
  /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/mlflow/utils/autologging_utils.py:60: DeprecationWarning: inspect.getargspec() is deprecated since Pytho
  n 3.0, use inspect.signature() or inspect.getfullargspec()
    all_param_names, _, _, all_default_values = inspect.getargspec(fn)  # pylint: disable=W1505
  Epoch 1/2
  /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:347: DeprecationWarning: Using or importing the ABCs from
  'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working
    if not isinstance(values, collections.Sequence):
    1/400 [..............................] - ETA: 0s - loss: 0.7153 - tp: 66972.0000 - fp: 0.0000e+00 - tn: 0.0000e+00 - fn: 14948.0000 - precision: 1.0000 - recall: 0.8175 - accuracy: 0.8175 - auc: 0.0000e+0
  02020-05-31 14:05:34.823955: I tensorflow/core/profiler/lib/profiler_session.cc:163] Profiler session started.
  WARNING:tensorflow:From /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/ops/summary_ops_v2.py:1277: stop (from tensorflow.python.eager
  .profiler) is deprecated and will be removed after 2020-07-01.
  Instructions for updating:
  use `tf.profiler.experimental.stop` instead.
  2020-05-31 14:05:36.334154: I tensorflow/core/profiler/rpc/client/save_profile.cc:176] Creating directory: /tmp/tb/train/plugins/profile/2020_05_31_14_05_36
  2020-05-31 14:05:36.348656: I tensorflow/core/profiler/rpc/client/save_profile.cc:182] Dumped gzipped tool data for trace.json.gz to /tmp/tb/train/plugins/profile/2020_05_31_14_05_36/node171.trace.json.gz
  2020-05-31 14:05:36.375326: I tensorflow/core/profiler/rpc/client/save_profile.cc:176] Creating directory: /tmp/tb/train/plugins/profile/2020_05_31_14_05_36
  2020-05-31 14:05:36.375452: I tensorflow/core/profiler/rpc/client/save_profile.cc:182] Dumped gzipped tool data for memory_profile.json.gz to /tmp/tb/train/plugins/profile/2020_05_31_14_05_36/node171.memory
  _profile.json.gz
  2020-05-31 14:05:36.377489: I tensorflow/python/profiler/internal/profiler_wrapper.cc:111] Creating directory: /tmp/tb/train/plugins/profile/2020_05_31_14_05_36Dumped tool data for xplane.pb to /tmp/tb/trai
  n/plugins/profile/2020_05_31_14_05_36/node171.xplane.pb
  Dumped tool data for overview_page.pb to /tmp/tb/train/plugins/profile/2020_05_31_14_05_36/node171.overview_page.pb
  Dumped tool data for input_pipeline.pb to /tmp/tb/train/plugins/profile/2020_05_31_14_05_36/node171.input_pipeline.pb
  Dumped tool data for tensorflow_stats.pb to /tmp/tb/train/plugins/profile/2020_05_31_14_05_36/node171.tensorflow_stats.pb
  Dumped tool data for kernel_stats.pb to /tmp/tb/train/plugins/profile/2020_05_31_14_05_36/node171.kernel_stats.pb

  400/400 [==============================] - 651s 2s/step - loss: 0.0019 - tp: 32753052.0000 - fp: 0.0000e+00 - tn: 0.0000e+00 - fn: 14948.0000 - precision: 1.0000 - recall: 0.9995 - accuracy: 0.9995 - auc: 0
  .0000e+00 - val_loss: 7.3428e-23 - val_tp: 16384000.0000 - val_fp: 0.0000e+00 - val_tn: 0.0000e+00 - val_fn: 0.0000e+00 - val_precision: 1.0000 - val_recall: 1.0000 - val_accuracy: 1.0000 - val_auc: 0.0000e
  +00
  Epoch 2/2
  400/400 [==============================] - 645s 2s/step - loss: 2.1658e-09 - tp: 32768000.0000 - fp: 0.0000e+00 - tn: 0.0000e+00 - fn: 0.0000e+00 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - au
  c: 0.0000e+00 - val_loss: 8.7165e-23 - val_tp: 16384000.0000 - val_fp: 0.0000e+00 - val_tn: 0.0000e+00 - val_fn: 0.0000e+00 - val_precision: 1.0000 - val_recall: 1.0000 - val_accuracy: 1.0000 - val_auc: 0.0
  000e+00
  400/400 [==============================] - 119s 298ms/step - loss: 9.0078e-23 - tp: 32768000.0000 - fp: 0.0000e+00 - tn: 0.0000e+00 - fn: 0.0000e+00 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 -
   auc: 0.0000e+00
  2020/05/31 14:29:15 INFO mlflow.projects: === Run (ID 'b37371694a004638a6cd7a94d4d2e77f') succeeded ===
  (tensorflow_nightly) [ye53nis@node171 drmed-git]$ Regard

mlflow ui

(tensorflow_nightly) [ye53nis@node171 drmed-git]$ mlflow ui
[2020-05-31 15:27:54 +0200] [200865] [INFO] Starting gunicorn 20.0.4
[2020-05-31 15:27:54 +0200] [200865] [INFO] Listening at: http://127.0.0.1:5000 (200865)
[2020-05-31 15:27:54 +0200] [200865] [INFO] Using worker: sync
[2020-05-31 15:27:54 +0200] [200871] [INFO] Booting worker with pid: 200871

mlflow run . -P fluotracify_path=/beegfs/ye53nis/drmed-git/src/ -P epochs=2 -P learning_rate=None -P csv_path=/beegfs/ye53nis/saves/firstartifact_Mar2020 -P steps_per_epoch=10 -P validation_steps=10

(tensorflow_nightly) [ye53nis@node171 drmed-git]$ mlflow run . -P fluotracify_path=/beegfs/ye53nis/drmed-git/src/ -P epochs=2 -P learning_rate=None -P csv_path=/beegfs/ye53nis/saves/firstartifact_Mar2020 -P
 steps_per_epoch=10 -P validation_steps=10
INFO: 'exp-310520-unet' does not exist. Creating a new experiment
2020/05/31 15:45:30 INFO mlflow.projects: === Created directory /tmp/tmp2vv5kiic for downloading remote URIs passed to arguments of type 'path' ===
2020/05/31 15:45:30 INFO mlflow.projects: === Running command 'source /cluster/miniconda3/bin/../etc/profile.d/conda.sh && conda activate mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b 1>&2 && python src/f
luotracify/training/train.py /beegfs/ye53nis/drmed-git/src 5 0.2 16384 None 2 /beegfs/ye53nis/saves/firstartifact_Mar2020 10 10' in run with ID '7f23f6ba7a244914b3cbbebd731d50a1' ===
2020-05-31 15:45:31.886019: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No
 such file or directory
2020-05-31 15:45:31.886068: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
/home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/framework/load_library.py:23: DeprecationWarning: the imp module is deprecated in favo
ur of importlib; see the module's documentation for alternative uses
  import imp
2.3.0-dev20200527
2020-05-31 15:45:35.151500: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file
 or directory
2020-05-31 15:45:35.151549: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: UNKNOWN ERROR (303)
2020-05-31 15:45:35.151589: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node171): /proc/driver/nvidia/version does not exist
GPUs:  []
train 0 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set027.csv
train 1 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set087.csv
train 2 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set003.csv
train 3 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set056.csv
train 4 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set076.csv
train 5 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set094.csv
train 6 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set017.csv
train 7 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set074.csv
train 8 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set055.csv
train 9 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set096.csv
train 10 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set054.csv
train 11 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set093.csv
train 12 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set079.csv
train 13 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set014.csv
train 14 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set008.csv
train 15 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set031.csv
train 16 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set023.csv
train 17 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set025.csv
train 18 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set034.csv
train 19 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set009.csv
train 20 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set044.csv
train 21 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set063.csv
train 22 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set004.csv
train 23 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set072.csv
train 24 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set046.csv
train 25 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set049.csv
train 26 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set007.csv
train 27 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set100.csv
train 28 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set083.csv
train 29 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set077.csv
train 30 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set061.csv
train 31 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set081.csv
train 32 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set091.csv
train 33 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set069.csv
train 34 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set052.csv
train 35 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set028.csv
train 36 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set019.csv
train 37 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set057.csv
train 38 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set064.csv
train 39 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set075.csv
train 40 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set002.csv
train 41 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set062.csv
train 42 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set043.csv
train 43 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set042.csv
train 44 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set005.csv
train 45 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set016.csv
train 46 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set018.csv
train 47 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set041.csv
train 48 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set039.csv
train 49 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set006.csv
train 50 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set092.csv
train 51 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set060.csv
train 52 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set001.csv
train 53 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set035.csv
train 54 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set029.csv
train 55 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set051.csv
train 56 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set012.csv
train 57 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set036.csv
train 58 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set024.csv
train 59 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set053.csv
train 60 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set011.csv
train 61 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set032.csv
train 62 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set067.csv
train 63 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set058.csv
train 64 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set080.csv
train 65 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set086.csv
train 66 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set033.csv
train 67 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set085.csv
train 68 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set015.csv
train 69 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set090.csv
train 70 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set020.csv
train 71 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set030.csv
train 72 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set050.csv
train 73 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set098.csv
train 74 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set099.csv
train 75 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set070.csv
train 76 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set021.csv
train 77 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set095.csv
train 78 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set073.csv
train 79 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set078.csv
test 80 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set026.csv
test 81 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set038.csv
test 82 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set082.csv
test 83 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set047.csv
test 84 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set040.csv
test 85 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set066.csv
test 86 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set059.csv
test 87 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set013.csv
test 88 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set089.csv
test 89 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set071.csv
test 90 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set088.csv
test 91 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set037.csv
test 92 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set022.csv
test 93 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set084.csv
test 94 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set010.csv
test 95 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set097.csv
test 96 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set068.csv
test 97 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set065.csv
test 98 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set048.csv
test 99 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set045.csv
shapes of feature dataframe: (16384, 8000) and label dataframe: (16384, 8000)
shapes of feature dataframe: (16384, 2000) and label dataframe: (16384, 2000)

for each 20,000 timestap trace there are the following numbers of corrupted timesteps:
 label001_1    16384
label001_1    16384
label001_1    16384
label001_1    16384
label001_1    16384
dtype: int64
2020-05-31 15:49:25.643578: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with Intel(R) MKL-DNN to use the following CPU instructions in performance-critical opera
tions:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-05-31 15:49:25.653635: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2300000000 Hz
2020-05-31 15:49:25.655125: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x556ab2694730 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-05-31 15:49:25.655169: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
number of training examples: 6400, number of validation examples: 1600

------------------------
number of test examples: 2000

input - shape:   (None, 16384, 1)
output - shape:  (None, 16384, 1)
2020-05-31 15:49:32.395226: I tensorflow/core/profiler/lib/profiler_session.cc:163] Profiler session started.
/home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/training/tracking/data_structures.py:739: DeprecationWarning: Using or importing the A
BCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working
  if not isinstance(wrapped_dict, collections.Mapping):
/home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/mlflow/utils/autologging_utils.py:60: DeprecationWarning: inspect.getargspec() is deprecated since Pytho
n 3.0, use inspect.signature() or inspect.getfullargspec()
  all_param_names, _, _, all_default_values = inspect.getargspec(fn)  # pylint: disable=W1505
Epoch 1/2
/home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:347: DeprecationWarning: Using or importing the ABCs from
'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working
  if not isinstance(values, collections.Sequence):
 1/10 [==>...........................] - ETA: 0s - loss: 0.5648 - tp: 72554.0000 - fp: 0.0000e+00 - tn: 0.0000e+00 - fn: 9366.0000 - precision: 1.0000 - recall: 0.8857 - accuracy: 0.8857 - auc: 0.0000e+0020
20-05-31 15:49:44.598769: I tensorflow/core/profiler/lib/profiler_session.cc:163] Profiler session started.
WARNING:tensorflow:From /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/ops/summary_ops_v2.py:1277: stop (from tensorflow.python.eager
.profiler) is deprecated and will be removed after 2020-07-01.
Instructions for updating:
use `tf.profiler.experimental.stop` instead.
2020-05-31 15:49:46.123954: I tensorflow/core/profiler/rpc/client/save_profile.cc:176] Creating directory: /tmp/tb/train/plugins/profile/2020_05_31_15_49_46
2020-05-31 15:49:46.137599: I tensorflow/core/profiler/rpc/client/save_profile.cc:182] Dumped gzipped tool data for trace.json.gz to /tmp/tb/train/plugins/profile/2020_05_31_15_49_46/node171.trace.json.gz
2020-05-31 15:49:46.163428: I tensorflow/core/profiler/rpc/client/save_profile.cc:176] Creating directory: /tmp/tb/train/plugins/profile/2020_05_31_15_49_46
2020-05-31 15:49:46.163531: I tensorflow/core/profiler/rpc/client/save_profile.cc:182] Dumped gzipped tool data for memory_profile.json.gz to /tmp/tb/train/plugins/profile/2020_05_31_15_49_46/node171.memory
_profile.json.gz
2020-05-31 15:49:46.165569: I tensorflow/python/profiler/internal/profiler_wrapper.cc:111] Creating directory: /tmp/tb/train/plugins/profile/2020_05_31_15_49_46Dumped tool data for xplane.pb to /tmp/tb/trai
n/plugins/profile/2020_05_31_15_49_46/node171.xplane.pb
Dumped tool data for overview_page.pb to /tmp/tb/train/plugins/profile/2020_05_31_15_49_46/node171.overview_page.pb
Dumped tool data for input_pipeline.pb to /tmp/tb/train/plugins/profile/2020_05_31_15_49_46/node171.input_pipeline.pb
Dumped tool data for tensorflow_stats.pb to /tmp/tb/train/plugins/profile/2020_05_31_15_49_46/node171.tensorflow_stats.pb
Dumped tool data for kernel_stats.pb to /tmp/tb/train/plugins/profile/2020_05_31_15_49_46/node171.kernel_stats.pb

10/10 [==============================] - 23s 2s/step - loss: 0.0638 - tp: 809834.0000 - fp: 0.0000e+00 - tn: 0.0000e+00 - fn: 9366.0000 - precision: 1.0000 - recall: 0.9886 - accuracy: 0.9886 - auc: 0.0000e
+00 - val_loss: 0.0000e+00 - val_tp: 819200.0000 - val_fp: 0.0000e+00 - val_tn: 0.0000e+00 - val_fn: 0.0000e+00 - val_precision: 1.0000 - val_recall: 1.0000 - val_accuracy: 1.0000 - val_auc: 0.0000e+00
Epoch 2/2
10/10 [==============================] - 18s 2s/step - loss: 6.4955e-17 - tp: 819200.0000 - fp: 0.0000e+00 - tn: 0.0000e+00 - fn: 0.0000e+00 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - auc: 0.
0000e+00 - val_loss: 0.0000e+00 - val_tp: 819200.0000 - val_fp: 0.0000e+00 - val_tn: 0.0000e+00 - val_fn: 0.0000e+00 - val_precision: 1.0000 - val_recall: 1.0000 - val_accuracy: 1.0000 - val_auc: 0.0000e+00
400/400 [==============================] - 120s 300ms/step - loss: 0.0000e+00 - tp: 32768000.0000 - fp: 0.0000e+00 - tn: 0.0000e+00 - fn: 0.0000e+00 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 -
 auc: 0.0000e+00
2020/05/31 15:52:36 INFO mlflow.projects: === Run (ID '7f23f6ba7a244914b3cbbebd731d50a1') succeeded ===

These two test runs used the wrong dataset! the firstartifact_Mar2020 dataset is bright clusters, but the label is the uncorrupted trace, not just the artifact information. This dataset is meant for a Variational Autoencoder-Training.

I started local machine:

import mlflow
import pprint
%cd /home/lex/Programme/drmed-git/

/home/lex/Programme/drmed-git

uri = 'file:///home/lex/Programme/drmed-git/data/mlruns'
mlflow.set_tracking_uri(uri)
runs = mlflow.search_runs(experiment_ids='0')
print('run_id of first in list: ', runs.iloc[0].run_id)
print('no of runs in list: ', len(runs))
print()
runs

run_id of first in list:  1aefda1366f04f5da5d1fc2241ad9208
no of runs in list:  1

client = mlflow.tracking.MlflowClient(tracking_uri=uri)

run_idx = 0

# mlflow.entities.Experiment
print('client.get_experiment()\n', exp.to_proto())
# mlflow.entities.Metric
# metent = client.get_metric_history(run_id=runs.iloc[run_idx].run_id, key='loss')
# for i in metent:
    # print(i)

print('- - - mlflow.entities.Run - - -')
run = client.get_run(runs.iloc[run_idx].run_id)
print('run.info.to_proto\n', run.info.to_proto())
pprint.pprint(run.data.metrics, sort_dicts=False)

model_path = client.download_artifacts(
    run_id=runs.iloc[run_idx].run_id,
    path='model')
tensorboard_path = client.download_artifacts(
    run_id=runs.iloc[run_idx].run_id,
    path='tensorboard_logs')



print('\nmodel_path\n', model_path)
print('\nMLmodel file')
%cat $model_path/MLmodel
print('\nconda.yaml')
%cat $model_path/conda.yaml
print('\nkeras_module.txt')
%cat $model_path/data/keras_module.txt

client.get_experiment()
 experiment_id: "0"
name: "exp-310520-unet"
artifact_location: "file:./data/mlruns/0"
lifecycle_stage: "active"

- - - mlflow.entities.Run - - -
run.info.to_proto
 run_uuid: "1aefda1366f04f5da5d1fc2241ad9208"
experiment_id: "0"
user_id: "ye53nis"
status: FINISHED
start_time: 1590953991905
end_time: 1590998871011
artifact_uri: "file:./data/mlruns/0/1aefda1366f04f5da5d1fc2241ad9208/artifacts"
lifecycle_stage: "active"
run_id: "1aefda1366f04f5da5d1fc2241ad9208"

{'learning rate': 1e-05,
 'val_loss': 0.3504984378814697,
 'precision': 0.9042773246765137,
 'fp': 711970.0,
 'loss': 0.2996121346950531,
 'val_tp': 3050870.0,
 'recall': 0.8978734016418457,
 'accuracy': 0.9549258947372437,
 'val_precision': 0.9677141904830933,
 'val_recall': 0.8017593622207642,
 'lr': 1e-05,
 'fn': 765019.0,
 'auc': 0.9786317348480225,
 'val_fp': 101786.0,
 'val_accuracy': 0.947745680809021,
 'val_tn': 12476995.0,
 'val_auc': 0.977130651473999,
 'val_fn': 754349.0,
 'tn': 24565164.0,
 'tp': 6725869.0}

model_path
 /home/lex/Programme/drmed-git/data/mlruns/0/1aefda1366f04f5da5d1fc2241ad9208/artifacts/model

MLmodel file
artifact_path: model
flavors:
  keras:
    data: data
    keras_module: tensorflow.keras
    keras_version: 2.2.4-tf
  python_function:
    data: data
    env: conda.yaml
    loader_module: mlflow.keras
    python_version: 3.8.3
run_id: 1aefda1366f04f5da5d1fc2241ad9208
utc_time_created: '2020-06-01 08:05:46.358978'

conda.yaml
channels:
- defaults
dependencies:
- python=3.8.3
- pip
- pip:
  - mlflow
  - tensorflow==2.3.0-dev20200527
name: mlflow-env

keras_module.txt
tensorflow.keras

summary_path
 /home/lex/Programme/drmed-git/data/mlruns/0/1aefda1366f04f5da5d1fc2241ad9208/artifacts/model_summary.txt

tensorboard_path
 /home/lex/Programme/drmed-git/data/mlruns/0/1aefda1366f04f5da5d1fc2241ad9208/artifacts/tensorboard_logs

summary_path = client.download_artifacts(
    run_id=runs.iloc[run_idx].run_id,
    path='model_summary.txt')

print('summary_path\n', summary_path, '\n')
%cat $summary_path

summary_path
 /home/lex/Programme/drmed-git/data/mlruns/0/1aefda1366f04f5da5d1fc2241ad9208/artifacts/model_summary.txt

Model: "functional_1"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to
==================================================================================================
input_1 (InputLayer)            [(None, 16384, 1)]   0
__________________________________________________________________________________________________
encode0 (Sequential)            (None, 16384, 64)    13120       input_1[0][0]
__________________________________________________________________________________________________
mp_encode0 (MaxPooling1D)       (None, 8192, 64)     0           encode0[0][0]
__________________________________________________________________________________________________
encode1 (Sequential)            (None, 8192, 128)    75008       mp_encode0[0][0]
__________________________________________________________________________________________________
mp_encode1 (MaxPooling1D)       (None, 4096, 128)    0           encode1[0][0]
__________________________________________________________________________________________________
encode2 (Sequential)            (None, 4096, 256)    297472      mp_encode1[0][0]
__________________________________________________________________________________________________
mp_encode2 (MaxPooling1D)       (None, 2048, 256)    0           encode2[0][0]
__________________________________________________________________________________________________
encode3 (Sequential)            (None, 2048, 512)    1184768     mp_encode2[0][0]
__________________________________________________________________________________________________
mp_encode3 (MaxPooling1D)       (None, 1024, 512)    0           encode3[0][0]
__________________________________________________________________________________________________
encode4 (Sequential)            (None, 1024, 512)    1577984     mp_encode3[0][0]
__________________________________________________________________________________________________
mp_encode4 (MaxPooling1D)       (None, 512, 512)     0           encode4[0][0]
__________________________________________________________________________________________________
encode5 (Sequential)            (None, 512, 512)     1577984     mp_encode4[0][0]
__________________________________________________________________________________________________
mp_encode5 (MaxPooling1D)       (None, 256, 512)     0           encode5[0][0]
__________________________________________________________________________________________________
encode6 (Sequential)            (None, 256, 512)     1577984     mp_encode5[0][0]
__________________________________________________________________________________________________
mp_encode6 (MaxPooling1D)       (None, 128, 512)     0           encode6[0][0]
__________________________________________________________________________________________________
encode7 (Sequential)            (None, 128, 512)     1577984     mp_encode6[0][0]
__________________________________________________________________________________________________
mp_encode7 (MaxPooling1D)       (None, 64, 512)      0           encode7[0][0]
__________________________________________________________________________________________________
encode8 (Sequential)            (None, 64, 512)      1577984     mp_encode7[0][0]
__________________________________________________________________________________________________
mp_encode8 (MaxPooling1D)       (None, 32, 512)      0           encode8[0][0]
__________________________________________________________________________________________________
two_conv_center (Sequential)    (None, 32, 1024)     4728832     mp_encode8[0][0]
__________________________________________________________________________________________________
conv_transpose_decoder8 (Sequen (None, 64, 512)      1051136     two_conv_center[0][0]
__________________________________________________________________________________________________
decoder8 (Concatenate)          (None, 64, 1024)     0           encode8[0][0]
                                                                 conv_transpose_decoder8[0][0]
__________________________________________________________________________________________________
two_conv_decoder8 (Sequential)  (None, 64, 512)      2364416     decoder8[0][0]
__________________________________________________________________________________________________
conv_transpose_decoder7 (Sequen (None, 128, 512)     526848      two_conv_decoder8[0][0]
__________________________________________________________________________________________________
decoder7 (Concatenate)          (None, 128, 1024)    0           encode7[0][0]
                                                                 conv_transpose_decoder7[0][0]
__________________________________________________________________________________________________
two_conv_decoder7 (Sequential)  (None, 128, 512)     2364416     decoder7[0][0]
__________________________________________________________________________________________________
conv_transpose_decoder6 (Sequen (None, 256, 512)     526848      two_conv_decoder7[0][0]
__________________________________________________________________________________________________
decoder6 (Concatenate)          (None, 256, 1024)    0           encode6[0][0]
                                                                 conv_transpose_decoder6[0][0]
__________________________________________________________________________________________________
two_conv_decoder6 (Sequential)  (None, 256, 512)     2364416     decoder6[0][0]
__________________________________________________________________________________________________
conv_transpose_decoder5 (Sequen (None, 512, 512)     526848      two_conv_decoder6[0][0]
__________________________________________________________________________________________________
decoder5 (Concatenate)          (None, 512, 1024)    0           encode5[0][0]
                                                                 conv_transpose_decoder5[0][0]
__________________________________________________________________________________________________
two_conv_decoder5 (Sequential)  (None, 512, 512)     2364416     decoder5[0][0]
__________________________________________________________________________________________________
conv_transpose_decoder4 (Sequen (None, 1024, 512)    526848      two_conv_decoder5[0][0]
__________________________________________________________________________________________________
decoder4 (Concatenate)          (None, 1024, 1024)   0           encode4[0][0]
                                                                 conv_transpose_decoder4[0][0]
__________________________________________________________________________________________________
two_conv_decoder4 (Sequential)  (None, 1024, 512)    2364416     decoder4[0][0]
__________________________________________________________________________________________________
conv_transpose_decoder3 (Sequen (None, 2048, 512)    526848      two_conv_decoder4[0][0]
__________________________________________________________________________________________________
decoder3 (Concatenate)          (None, 2048, 1024)   0           encode3[0][0]
                                                                 conv_transpose_decoder3[0][0]
__________________________________________________________________________________________________
two_conv_decoder3 (Sequential)  (None, 2048, 512)    2364416     decoder3[0][0]
__________________________________________________________________________________________________
conv_transpose_decoder2 (Sequen (None, 4096, 256)    263424      two_conv_decoder3[0][0]
__________________________________________________________________________________________________
decoder2 (Concatenate)          (None, 4096, 512)    0           encode2[0][0]
                                                                 conv_transpose_decoder2[0][0]
__________________________________________________________________________________________________
two_conv_decoder2 (Sequential)  (None, 4096, 256)    592384      decoder2[0][0]
__________________________________________________________________________________________________
conv_transpose_decoder1 (Sequen (None, 8192, 128)    66176       two_conv_decoder2[0][0]
__________________________________________________________________________________________________
decoder1 (Concatenate)          (None, 8192, 256)    0           encode1[0][0]
                                                                 conv_transpose_decoder1[0][0]
__________________________________________________________________________________________________
two_conv_decoder1 (Sequential)  (None, 8192, 128)    148736      decoder1[0][0]
__________________________________________________________________________________________________
conv_transpose_decoder0 (Sequen (None, 16384, 64)    16704       two_conv_decoder1[0][0]
__________________________________________________________________________________________________
decoder0 (Concatenate)          (None, 16384, 128)   0           encode0[0][0]
                                                                 conv_transpose_decoder0[0][0]
__________________________________________________________________________________________________
two_conv_decoder0 (Sequential)  (None, 16384, 64)    37504       decoder0[0][0]
__________________________________________________________________________________________________
conv1d_38 (Conv1D)              (None, 16384, 1)     65          two_conv_decoder0[0][0]
==================================================================================================
Total params: 33,185,985
Trainable params: 33,146,689
Non-trainable params: 39,296
__________________________________________________________________________________________________

  print('\ntensorboard_path\n', tensorboard_path)
  !ls $tensorboard_path

# https://stackoverflow.com/questions/41074688/how-do-you-read-tensorboard-files-programmatically
from tensorboard.backend.event_processing import event_accumulator
path = str(tensorboard_path) + '/train'
path2 = str()
print(path)
ea = event_accumulator.EventAccumulator(path=path,
    size_guidance={ # see below regarding this argument
        event_accumulator.COMPRESSED_HISTOGRAMS: 500,
        event_accumulator.IMAGES: 4,
        event_accumulator.AUDIO: 4,
        event_accumulator.SCALARS: 0,
        event_accumulator.HISTOGRAMS: 1,
    })

ea.Reload() # loads events from file

tensorboard_path
 /home/lex/Programme/drmed-git/data/mlruns/0/1aefda1366f04f5da5d1fc2241ad9208/artifacts/tensorboard_logs
image  metrics	train  validation
/home/lex/Programme/drmed-git/data/mlruns/0/1aefda1366f04f5da5d1fc2241ad9208/artifacts/tensorboard_logs/train
<tensorboard.backend.event_processing.event_accumulator.EventAccumulator at 0x7fdb70770c70>

ea.Tags()

{'images': ['encode0/conv1d/kernel_0/image/0',
    'encode0/conv1d/bias_0/image/0',
    'encode0/batch_normalization/gamma_0/image/0',
    'encode0/batch_normalization/beta_0/image/0',
    'encode0/batch_normalization/moving_mean_0/image/0',
    'encode0/batch_normalization/moving_variance_0/image/0',
    'encode0/conv1d_1/kernel_0/image/0',
    'encode0/conv1d_1/kernel_0/image/1',
    'encode0/conv1d_1/kernel_0/image/2',
    'encode0/conv1d_1/bias_0/image/0',
    'encode0/batch_normalization_1/gamma_0/image/0',
    'encode0/batch_normalization_1/beta_0/image/0',
    'encode0/batch_normalization_1/moving_mean_0/image/0',
    'encode0/batch_normalization_1/moving_variance_0/image/0',
    'encode1/conv1d_2/kernel_0/image/0',
    'encode1/conv1d_2/kernel_0/image/1',
    'encode1/conv1d_2/kernel_0/image/2',
    'encode1/conv1d_2/bias_0/image/0',
    'encode1/batch_normalization_2/gamma_0/image/0',
    'encode1/batch_normalization_2/beta_0/image/0',
    'encode1/batch_normalization_2/moving_mean_0/image/0',
    'encode1/batch_normalization_2/moving_variance_0/image/0',
    'encode1/conv1d_3/kernel_0/image/0',
    'encode1/conv1d_3/kernel_0/image/1',
    'encode1/conv1d_3/kernel_0/image/2',
    'encode1/conv1d_3/bias_0/image/0',
    'encode1/batch_normalization_3/gamma_0/image/0',
    'encode1/batch_normalization_3/beta_0/image/0',
    'encode1/batch_normalization_3/moving_mean_0/image/0',
    'encode1/batch_normalization_3/moving_variance_0/image/0',
    'encode2/conv1d_4/kernel_0/image/0',
    'encode2/conv1d_4/kernel_0/image/1',
    'encode2/conv1d_4/kernel_0/image/2',
    'encode2/conv1d_4/bias_0/image/0',
    'encode2/batch_normalization_4/gamma_0/image/0',
    'encode2/batch_normalization_4/beta_0/image/0',
    'encode2/batch_normalization_4/moving_mean_0/image/0',
    'encode2/batch_normalization_4/moving_variance_0/image/0',
    'encode2/conv1d_5/kernel_0/image/0',
    'encode2/conv1d_5/kernel_0/image/1',
    'encode2/conv1d_5/kernel_0/image/2',
    'encode2/conv1d_5/bias_0/image/0',
    'encode2/batch_normalization_5/gamma_0/image/0',
    'encode2/batch_normalization_5/beta_0/image/0',
    'encode2/batch_normalization_5/moving_mean_0/image/0',
    'encode2/batch_normalization_5/moving_variance_0/image/0',
    'encode3/conv1d_6/kernel_0/image/0',
    'encode3/conv1d_6/kernel_0/image/1',
    'encode3/conv1d_6/kernel_0/image/2',
    'encode3/conv1d_6/bias_0/image/0',
    'encode3/batch_normalization_6/gamma_0/image/0',
    'encode3/batch_normalization_6/beta_0/image/0',
    'encode3/batch_normalization_6/moving_mean_0/image/0',
    'encode3/batch_normalization_6/moving_variance_0/image/0',
    'encode3/conv1d_7/kernel_0/image/0',
    'encode3/conv1d_7/kernel_0/image/1',
    'encode3/conv1d_7/kernel_0/image/2',
    'encode3/conv1d_7/bias_0/image/0',
    'encode3/batch_normalization_7/gamma_0/image/0',
    'encode3/batch_normalization_7/beta_0/image/0',
    'encode3/batch_normalization_7/moving_mean_0/image/0',
    'encode3/batch_normalization_7/moving_variance_0/image/0',
    'encode4/conv1d_8/kernel_0/image/0',
    'encode4/conv1d_8/kernel_0/image/1',
    'encode4/conv1d_8/kernel_0/image/2',
    'encode4/conv1d_8/bias_0/image/0',
    'encode4/batch_normalization_8/gamma_0/image/0',
    'encode4/batch_normalization_8/beta_0/image/0',
    'encode4/batch_normalization_8/moving_mean_0/image/0',
    'encode4/batch_normalization_8/moving_variance_0/image/0',
    'encode4/conv1d_9/kernel_0/image/0',
    'encode4/conv1d_9/kernel_0/image/1',
    'encode4/conv1d_9/kernel_0/image/2',
    'encode4/conv1d_9/bias_0/image/0',
    'encode4/batch_normalization_9/gamma_0/image/0',
    'encode4/batch_normalization_9/beta_0/image/0',
    'encode4/batch_normalization_9/moving_mean_0/image/0',
    'encode4/batch_normalization_9/moving_variance_0/image/0',
    'encode5/conv1d_10/kernel_0/image/0',
    'encode5/conv1d_10/kernel_0/image/1',
    'encode5/conv1d_10/kernel_0/image/2',
    'encode5/conv1d_10/bias_0/image/0',
    'encode5/batch_normalization_10/gamma_0/image/0',
    'encode5/batch_normalization_10/beta_0/image/0',
    'encode5/batch_normalization_10/moving_mean_0/image/0',
    'encode5/batch_normalization_10/moving_variance_0/image/0',
    'encode5/conv1d_11/kernel_0/image/0',
    'encode5/conv1d_11/kernel_0/image/1',
    'encode5/conv1d_11/kernel_0/image/2',
    'encode5/conv1d_11/bias_0/image/0',
    'encode5/batch_normalization_11/gamma_0/image/0',
    'encode5/batch_normalization_11/beta_0/image/0',
    'encode5/batch_normalization_11/moving_mean_0/image/0',
    'encode5/batch_normalization_11/moving_variance_0/image/0',
    'encode6/conv1d_12/kernel_0/image/0',
    'encode6/conv1d_12/kernel_0/image/1',
    'encode6/conv1d_12/kernel_0/image/2',
    'encode6/conv1d_12/bias_0/image/0',
    'encode6/batch_normalization_12/gamma_0/image/0',
    'encode6/batch_normalization_12/beta_0/image/0',
    'encode6/batch_normalization_12/moving_mean_0/image/0',
    'encode6/batch_normalization_12/moving_variance_0/image/0',
    'encode6/conv1d_13/kernel_0/image/0',
    'encode6/conv1d_13/kernel_0/image/1',
    'encode6/conv1d_13/kernel_0/image/2',
    'encode6/conv1d_13/bias_0/image/0',
    'encode6/batch_normalization_13/gamma_0/image/0',
    'encode6/batch_normalization_13/beta_0/image/0',
    'encode6/batch_normalization_13/moving_mean_0/image/0',
    'encode6/batch_normalization_13/moving_variance_0/image/0',
    'encode7/conv1d_14/kernel_0/image/0',
    'encode7/conv1d_14/kernel_0/image/1',
    'encode7/conv1d_14/kernel_0/image/2',
    'encode7/conv1d_14/bias_0/image/0',
    'encode7/batch_normalization_14/gamma_0/image/0',
    'encode7/batch_normalization_14/beta_0/image/0',
    'encode7/batch_normalization_14/moving_mean_0/image/0',
    'encode7/batch_normalization_14/moving_variance_0/image/0',
    'encode7/conv1d_15/kernel_0/image/0',
    'encode7/conv1d_15/kernel_0/image/1',
    'encode7/conv1d_15/kernel_0/image/2',
    'encode7/conv1d_15/bias_0/image/0',
    'encode7/batch_normalization_15/gamma_0/image/0',
    'encode7/batch_normalization_15/beta_0/image/0',
    'encode7/batch_normalization_15/moving_mean_0/image/0',
    'encode7/batch_normalization_15/moving_variance_0/image/0',
    'encode8/conv1d_16/kernel_0/image/0',
    'encode8/conv1d_16/kernel_0/image/1',
    'encode8/conv1d_16/kernel_0/image/2',
    'encode8/conv1d_16/bias_0/image/0',
    'encode8/batch_normalization_16/gamma_0/image/0',
    'encode8/batch_normalization_16/beta_0/image/0',
    'encode8/batch_normalization_16/moving_mean_0/image/0',
    'encode8/batch_normalization_16/moving_variance_0/image/0',
    'encode8/conv1d_17/kernel_0/image/0',
    'encode8/conv1d_17/kernel_0/image/1',
    'encode8/conv1d_17/kernel_0/image/2',
    'encode8/conv1d_17/bias_0/image/0',
    'encode8/batch_normalization_17/gamma_0/image/0',
    'encode8/batch_normalization_17/beta_0/image/0',
    'encode8/batch_normalization_17/moving_mean_0/image/0',
    'encode8/batch_normalization_17/moving_variance_0/image/0',
    'two_conv_center/conv1d_18/kernel_0/image/0',
    'two_conv_center/conv1d_18/kernel_0/image/1',
    'two_conv_center/conv1d_18/kernel_0/image/2',
    'two_conv_center/conv1d_18/bias_0/image/0',
    'two_conv_center/batch_normalization_18/gamma_0/image/0',
    'two_conv_center/batch_normalization_18/beta_0/image/0',
    'two_conv_center/batch_normalization_18/moving_mean_0/image/0',
    'two_conv_center/batch_normalization_18/moving_variance_0/image/0',
    'two_conv_center/conv1d_19/kernel_0/image/0',
    'two_conv_center/conv1d_19/kernel_0/image/1',
    'two_conv_center/conv1d_19/kernel_0/image/2',
    'two_conv_center/conv1d_19/bias_0/image/0',
    'two_conv_center/batch_normalization_19/gamma_0/image/0',
    'two_conv_center/batch_normalization_19/beta_0/image/0',
    'two_conv_center/batch_normalization_19/moving_mean_0/image/0',
    'two_conv_center/batch_normalization_19/moving_variance_0/image/0',
    'conv_transpose_decoder8/conv1d_transpose/kernel_0/image/0',
    'conv_transpose_decoder8/conv1d_transpose/kernel_0/image/1',
    'conv_transpose_decoder8/conv1d_transpose/kernel_0/image/2',
    'conv_transpose_decoder8/conv1d_transpose/bias_0/image/0',
    'conv_transpose_decoder8/batch_normalization_20/gamma_0/image/0',
    'conv_transpose_decoder8/batch_normalization_20/beta_0/image/0',
    'conv_transpose_decoder8/batch_normalization_20/moving_mean_0/image/0',
    'conv_transpose_decoder8/batch_normalization_20/moving_variance_0/image/0',
    'two_conv_decoder8/conv1d_20/kernel_0/image/0',
    'two_conv_decoder8/conv1d_20/kernel_0/image/1',
    'two_conv_decoder8/conv1d_20/kernel_0/image/2',
    'two_conv_decoder8/conv1d_20/bias_0/image/0',
    'two_conv_decoder8/batch_normalization_21/gamma_0/image/0',
    'two_conv_decoder8/batch_normalization_21/beta_0/image/0',
    'two_conv_decoder8/batch_normalization_21/moving_mean_0/image/0',
    'two_conv_decoder8/batch_normalization_21/moving_variance_0/image/0',
    'two_conv_decoder8/conv1d_21/kernel_0/image/0',
    'two_conv_decoder8/conv1d_21/kernel_0/image/1',
    'two_conv_decoder8/conv1d_21/kernel_0/image/2',
    'two_conv_decoder8/conv1d_21/bias_0/image/0',
    'two_conv_decoder8/batch_normalization_22/gamma_0/image/0',
    'two_conv_decoder8/batch_normalization_22/beta_0/image/0',
    'two_conv_decoder8/batch_normalization_22/moving_mean_0/image/0',
    'two_conv_decoder8/batch_normalization_22/moving_variance_0/image/0',
    'conv_transpose_decoder7/conv1d_transpose_1/kernel_0/image/0',
    'conv_transpose_decoder7/conv1d_transpose_1/kernel_0/image/1',
    'conv_transpose_decoder7/conv1d_transpose_1/kernel_0/image/2',
    'conv_transpose_decoder7/conv1d_transpose_1/bias_0/image/0',
    'conv_transpose_decoder7/batch_normalization_23/gamma_0/image/0',
    'conv_transpose_decoder7/batch_normalization_23/beta_0/image/0',
    'conv_transpose_decoder7/batch_normalization_23/moving_mean_0/image/0',
    'conv_transpose_decoder7/batch_normalization_23/moving_variance_0/image/0',
    'two_conv_decoder7/conv1d_22/kernel_0/image/0',
    'two_conv_decoder7/conv1d_22/kernel_0/image/1',
    'two_conv_decoder7/conv1d_22/kernel_0/image/2',
    'two_conv_decoder7/conv1d_22/bias_0/image/0',
    'two_conv_decoder7/batch_normalization_24/gamma_0/image/0',
    'two_conv_decoder7/batch_normalization_24/beta_0/image/0',
    'two_conv_decoder7/batch_normalization_24/moving_mean_0/image/0',
    'two_conv_decoder7/batch_normalization_24/moving_variance_0/image/0',
    'two_conv_decoder7/conv1d_23/kernel_0/image/0',
    'two_conv_decoder7/conv1d_23/kernel_0/image/1',
    'two_conv_decoder7/conv1d_23/kernel_0/image/2',
    'two_conv_decoder7/conv1d_23/bias_0/image/0',
    'two_conv_decoder7/batch_normalization_25/gamma_0/image/0',
    'two_conv_decoder7/batch_normalization_25/beta_0/image/0',
    'two_conv_decoder7/batch_normalization_25/moving_mean_0/image/0',
    'two_conv_decoder7/batch_normalization_25/moving_variance_0/image/0',
    'conv_transpose_decoder6/conv1d_transpose_2/kernel_0/image/0',
    'conv_transpose_decoder6/conv1d_transpose_2/kernel_0/image/1',
    'conv_transpose_decoder6/conv1d_transpose_2/kernel_0/image/2',
    'conv_transpose_decoder6/conv1d_transpose_2/bias_0/image/0',
    'conv_transpose_decoder6/batch_normalization_26/gamma_0/image/0',
    'conv_transpose_decoder6/batch_normalization_26/beta_0/image/0',
    'conv_transpose_decoder6/batch_normalization_26/moving_mean_0/image/0',
    'conv_transpose_decoder6/batch_normalization_26/moving_variance_0/image/0',
    'two_conv_decoder6/conv1d_24/kernel_0/image/0',
    'two_conv_decoder6/conv1d_24/kernel_0/image/1',
    'two_conv_decoder6/conv1d_24/kernel_0/image/2',
    'two_conv_decoder6/conv1d_24/bias_0/image/0',
    'two_conv_decoder6/batch_normalization_27/gamma_0/image/0',
    'two_conv_decoder6/batch_normalization_27/beta_0/image/0',
    'two_conv_decoder6/batch_normalization_27/moving_mean_0/image/0',
    'two_conv_decoder6/batch_normalization_27/moving_variance_0/image/0',
    'two_conv_decoder6/conv1d_25/kernel_0/image/0',
    'two_conv_decoder6/conv1d_25/kernel_0/image/1',
    'two_conv_decoder6/conv1d_25/kernel_0/image/2',
    'two_conv_decoder6/conv1d_25/bias_0/image/0',
    'two_conv_decoder6/batch_normalization_28/gamma_0/image/0',
    'two_conv_decoder6/batch_normalization_28/beta_0/image/0',
    'two_conv_decoder6/batch_normalization_28/moving_mean_0/image/0',
    'two_conv_decoder6/batch_normalization_28/moving_variance_0/image/0',
    'conv_transpose_decoder5/conv1d_transpose_3/kernel_0/image/0',
    'conv_transpose_decoder5/conv1d_transpose_3/kernel_0/image/1',
    'conv_transpose_decoder5/conv1d_transpose_3/kernel_0/image/2',
    'conv_transpose_decoder5/conv1d_transpose_3/bias_0/image/0',
    'conv_transpose_decoder5/batch_normalization_29/gamma_0/image/0',
    'conv_transpose_decoder5/batch_normalization_29/beta_0/image/0',
    'conv_transpose_decoder5/batch_normalization_29/moving_mean_0/image/0',
    'conv_transpose_decoder5/batch_normalization_29/moving_variance_0/image/0',
    'two_conv_decoder5/conv1d_26/kernel_0/image/0',
    'two_conv_decoder5/conv1d_26/kernel_0/image/1',
    'two_conv_decoder5/conv1d_26/kernel_0/image/2',
    'two_conv_decoder5/conv1d_26/bias_0/image/0',
    'two_conv_decoder5/batch_normalization_30/gamma_0/image/0',
    'two_conv_decoder5/batch_normalization_30/beta_0/image/0',
    'two_conv_decoder5/batch_normalization_30/moving_mean_0/image/0',
    'two_conv_decoder5/batch_normalization_30/moving_variance_0/image/0',
    'two_conv_decoder5/conv1d_27/kernel_0/image/0',
    'two_conv_decoder5/conv1d_27/kernel_0/image/1',
    'two_conv_decoder5/conv1d_27/kernel_0/image/2',
    'two_conv_decoder5/conv1d_27/bias_0/image/0',
    'two_conv_decoder5/batch_normalization_31/gamma_0/image/0',
    'two_conv_decoder5/batch_normalization_31/beta_0/image/0',
    'two_conv_decoder5/batch_normalization_31/moving_mean_0/image/0',
    'two_conv_decoder5/batch_normalization_31/moving_variance_0/image/0',
    'conv_transpose_decoder4/conv1d_transpose_4/kernel_0/image/0',
    'conv_transpose_decoder4/conv1d_transpose_4/kernel_0/image/1',
    'conv_transpose_decoder4/conv1d_transpose_4/kernel_0/image/2',
    'conv_transpose_decoder4/conv1d_transpose_4/bias_0/image/0',
    'conv_transpose_decoder4/batch_normalization_32/gamma_0/image/0',
    'conv_transpose_decoder4/batch_normalization_32/beta_0/image/0',
    'conv_transpose_decoder4/batch_normalization_32/moving_mean_0/image/0',
    'conv_transpose_decoder4/batch_normalization_32/moving_variance_0/image/0',
    'two_conv_decoder4/conv1d_28/kernel_0/image/0',
    'two_conv_decoder4/conv1d_28/kernel_0/image/1',
    'two_conv_decoder4/conv1d_28/kernel_0/image/2',
    'two_conv_decoder4/conv1d_28/bias_0/image/0',
    'two_conv_decoder4/batch_normalization_33/gamma_0/image/0',
    'two_conv_decoder4/batch_normalization_33/beta_0/image/0',
    'two_conv_decoder4/batch_normalization_33/moving_mean_0/image/0',
    'two_conv_decoder4/batch_normalization_33/moving_variance_0/image/0',
    'two_conv_decoder4/conv1d_29/kernel_0/image/0',
    'two_conv_decoder4/conv1d_29/kernel_0/image/1',
    'two_conv_decoder4/conv1d_29/kernel_0/image/2',
    'two_conv_decoder4/conv1d_29/bias_0/image/0',
    'two_conv_decoder4/batch_normalization_34/gamma_0/image/0',
    'two_conv_decoder4/batch_normalization_34/beta_0/image/0',
    'two_conv_decoder4/batch_normalization_34/moving_mean_0/image/0',
    'two_conv_decoder4/batch_normalization_34/moving_variance_0/image/0',
    'conv_transpose_decoder3/conv1d_transpose_5/kernel_0/image/0',
    'conv_transpose_decoder3/conv1d_transpose_5/kernel_0/image/1',
    'conv_transpose_decoder3/conv1d_transpose_5/kernel_0/image/2',
    'conv_transpose_decoder3/conv1d_transpose_5/bias_0/image/0',
    'conv_transpose_decoder3/batch_normalization_35/gamma_0/image/0',
    'conv_transpose_decoder3/batch_normalization_35/beta_0/image/0',
    'conv_transpose_decoder3/batch_normalization_35/moving_mean_0/image/0',
    'conv_transpose_decoder3/batch_normalization_35/moving_variance_0/image/0',
    'two_conv_decoder3/conv1d_30/kernel_0/image/0',
    'two_conv_decoder3/conv1d_30/kernel_0/image/1',
    'two_conv_decoder3/conv1d_30/kernel_0/image/2',
    'two_conv_decoder3/conv1d_30/bias_0/image/0',
    'two_conv_decoder3/batch_normalization_36/gamma_0/image/0',
    'two_conv_decoder3/batch_normalization_36/beta_0/image/0',
    'two_conv_decoder3/batch_normalization_36/moving_mean_0/image/0',
    'two_conv_decoder3/batch_normalization_36/moving_variance_0/image/0',
    'two_conv_decoder3/conv1d_31/kernel_0/image/0',
    'two_conv_decoder3/conv1d_31/kernel_0/image/1',
    'two_conv_decoder3/conv1d_31/kernel_0/image/2',
    'two_conv_decoder3/conv1d_31/bias_0/image/0',
    'two_conv_decoder3/batch_normalization_37/gamma_0/image/0',
    'two_conv_decoder3/batch_normalization_37/beta_0/image/0',
    'two_conv_decoder3/batch_normalization_37/moving_mean_0/image/0',
    'two_conv_decoder3/batch_normalization_37/moving_variance_0/image/0',
    'conv_transpose_decoder2/conv1d_transpose_6/kernel_0/image/0',
    'conv_transpose_decoder2/conv1d_transpose_6/kernel_0/image/1',
    'conv_transpose_decoder2/conv1d_transpose_6/kernel_0/image/2',
    'conv_transpose_decoder2/conv1d_transpose_6/bias_0/image/0',
    'conv_transpose_decoder2/batch_normalization_38/gamma_0/image/0',
    'conv_transpose_decoder2/batch_normalization_38/beta_0/image/0',
    'conv_transpose_decoder2/batch_normalization_38/moving_mean_0/image/0',
    'conv_transpose_decoder2/batch_normalization_38/moving_variance_0/image/0',
    'two_conv_decoder2/conv1d_32/kernel_0/image/0',
    'two_conv_decoder2/conv1d_32/kernel_0/image/1',
    'two_conv_decoder2/conv1d_32/kernel_0/image/2',
    'two_conv_decoder2/conv1d_32/bias_0/image/0',
    'two_conv_decoder2/batch_normalization_39/gamma_0/image/0',
    'two_conv_decoder2/batch_normalization_39/beta_0/image/0',
    'two_conv_decoder2/batch_normalization_39/moving_mean_0/image/0',
    'two_conv_decoder2/batch_normalization_39/moving_variance_0/image/0',
    'two_conv_decoder2/conv1d_33/kernel_0/image/0',
    'two_conv_decoder2/conv1d_33/kernel_0/image/1',
    'two_conv_decoder2/conv1d_33/kernel_0/image/2',
    'two_conv_decoder2/conv1d_33/bias_0/image/0',
    'two_conv_decoder2/batch_normalization_40/gamma_0/image/0',
    'two_conv_decoder2/batch_normalization_40/beta_0/image/0',
    'two_conv_decoder2/batch_normalization_40/moving_mean_0/image/0',
    'two_conv_decoder2/batch_normalization_40/moving_variance_0/image/0',
    'conv_transpose_decoder1/conv1d_transpose_7/kernel_0/image/0',
    'conv_transpose_decoder1/conv1d_transpose_7/kernel_0/image/1',
    'conv_transpose_decoder1/conv1d_transpose_7/kernel_0/image/2',
    'conv_transpose_decoder1/conv1d_transpose_7/bias_0/image/0',
    'conv_transpose_decoder1/batch_normalization_41/gamma_0/image/0',
    'conv_transpose_decoder1/batch_normalization_41/beta_0/image/0',
    'conv_transpose_decoder1/batch_normalization_41/moving_mean_0/image/0',
    'conv_transpose_decoder1/batch_normalization_41/moving_variance_0/image/0',
    'two_conv_decoder1/conv1d_34/kernel_0/image/0',
    'two_conv_decoder1/conv1d_34/kernel_0/image/1',
    'two_conv_decoder1/conv1d_34/kernel_0/image/2',
    'two_conv_decoder1/conv1d_34/bias_0/image/0',
    'two_conv_decoder1/batch_normalization_42/gamma_0/image/0',
    'two_conv_decoder1/batch_normalization_42/beta_0/image/0',
    'two_conv_decoder1/batch_normalization_42/moving_mean_0/image/0',
    'two_conv_decoder1/batch_normalization_42/moving_variance_0/image/0',
    'two_conv_decoder1/conv1d_35/kernel_0/image/0',
    'two_conv_decoder1/conv1d_35/kernel_0/image/1',
    'two_conv_decoder1/conv1d_35/kernel_0/image/2',
    'two_conv_decoder1/conv1d_35/bias_0/image/0',
    'two_conv_decoder1/batch_normalization_43/gamma_0/image/0',
    'two_conv_decoder1/batch_normalization_43/beta_0/image/0',
    'two_conv_decoder1/batch_normalization_43/moving_mean_0/image/0',
    'two_conv_decoder1/batch_normalization_43/moving_variance_0/image/0',
    'conv_transpose_decoder0/conv1d_transpose_8/kernel_0/image/0',
    'conv_transpose_decoder0/conv1d_transpose_8/kernel_0/image/1',
    'conv_transpose_decoder0/conv1d_transpose_8/kernel_0/image/2',
    'conv_transpose_decoder0/conv1d_transpose_8/bias_0/image/0',
    'conv_transpose_decoder0/batch_normalization_44/gamma_0/image/0',
    'conv_transpose_decoder0/batch_normalization_44/beta_0/image/0',
    'conv_transpose_decoder0/batch_normalization_44/moving_mean_0/image/0',
    'conv_transpose_decoder0/batch_normalization_44/moving_variance_0/image/0',
    'two_conv_decoder0/conv1d_36/kernel_0/image/0',
    'two_conv_decoder0/conv1d_36/kernel_0/image/1',
    'two_conv_decoder0/conv1d_36/kernel_0/image/2',
    'two_conv_decoder0/conv1d_36/bias_0/image/0',
    'two_conv_decoder0/batch_normalization_45/gamma_0/image/0',
    'two_conv_decoder0/batch_normalization_45/beta_0/image/0',
    'two_conv_decoder0/batch_normalization_45/moving_mean_0/image/0',
    'two_conv_decoder0/batch_normalization_45/moving_variance_0/image/0',
    'two_conv_decoder0/conv1d_37/kernel_0/image/0',
    'two_conv_decoder0/conv1d_37/kernel_0/image/1',
    'two_conv_decoder0/conv1d_37/kernel_0/image/2',
    'two_conv_decoder0/conv1d_37/bias_0/image/0',
    'two_conv_decoder0/batch_normalization_46/gamma_0/image/0',
    'two_conv_decoder0/batch_normalization_46/beta_0/image/0',
    'two_conv_decoder0/batch_normalization_46/moving_mean_0/image/0',
    'two_conv_decoder0/batch_normalization_46/moving_variance_0/image/0',
    'conv1d_38/kernel_0/image/0'],
   'audio': [],
   'histograms': ['encode0/conv1d/kernel_0',
    'encode0/conv1d/bias_0',
    'encode0/batch_normalization/gamma_0',
    'encode0/batch_normalization/beta_0',
    'encode0/batch_normalization/moving_mean_0',
    'encode0/batch_normalization/moving_variance_0',
    'encode0/conv1d_1/kernel_0',
    'encode0/conv1d_1/bias_0',
    'encode0/batch_normalization_1/gamma_0',
    'encode0/batch_normalization_1/beta_0',
    'encode0/batch_normalization_1/moving_mean_0',
    'encode0/batch_normalization_1/moving_variance_0',
    'encode1/conv1d_2/kernel_0',
    'encode1/conv1d_2/bias_0',
    'encode1/batch_normalization_2/gamma_0',
    'encode1/batch_normalization_2/beta_0',
    'encode1/batch_normalization_2/moving_mean_0',
    'encode1/batch_normalization_2/moving_variance_0',
    'encode1/conv1d_3/kernel_0',
    'encode1/conv1d_3/bias_0',
    'encode1/batch_normalization_3/gamma_0',
    'encode1/batch_normalization_3/beta_0',
    'encode1/batch_normalization_3/moving_mean_0',
    'encode1/batch_normalization_3/moving_variance_0',
    'encode2/conv1d_4/kernel_0',
    'encode2/conv1d_4/bias_0',
    'encode2/batch_normalization_4/gamma_0',
    'encode2/batch_normalization_4/beta_0',
    'encode2/batch_normalization_4/moving_mean_0',
    'encode2/batch_normalization_4/moving_variance_0',
    'encode2/conv1d_5/kernel_0',
    'encode2/conv1d_5/bias_0',
    'encode2/batch_normalization_5/gamma_0',
    'encode2/batch_normalization_5/beta_0',
    'encode2/batch_normalization_5/moving_mean_0',
    'encode2/batch_normalization_5/moving_variance_0',
    'encode3/conv1d_6/kernel_0',
    'encode3/conv1d_6/bias_0',
    'encode3/batch_normalization_6/gamma_0',
    'encode3/batch_normalization_6/beta_0',
    'encode3/batch_normalization_6/moving_mean_0',
    'encode3/batch_normalization_6/moving_variance_0',
    'encode3/conv1d_7/kernel_0',
    'encode3/conv1d_7/bias_0',
    'encode3/batch_normalization_7/gamma_0',
    'encode3/batch_normalization_7/beta_0',
    'encode3/batch_normalization_7/moving_mean_0',
    'encode3/batch_normalization_7/moving_variance_0',
    'encode4/conv1d_8/kernel_0',
    'encode4/conv1d_8/bias_0',
    'encode4/batch_normalization_8/gamma_0',
    'encode4/batch_normalization_8/beta_0',
    'encode4/batch_normalization_8/moving_mean_0',
    'encode4/batch_normalization_8/moving_variance_0',
    'encode4/conv1d_9/kernel_0',
    'encode4/conv1d_9/bias_0',
    'encode4/batch_normalization_9/gamma_0',
    'encode4/batch_normalization_9/beta_0',
    'encode4/batch_normalization_9/moving_mean_0',
    'encode4/batch_normalization_9/moving_variance_0',
    'encode5/conv1d_10/kernel_0',
    'encode5/conv1d_10/bias_0',
    'encode5/batch_normalization_10/gamma_0',
    'encode5/batch_normalization_10/beta_0',
    'encode5/batch_normalization_10/moving_mean_0',
    'encode5/batch_normalization_10/moving_variance_0',
    'encode5/conv1d_11/kernel_0',
    'encode5/conv1d_11/bias_0',
    'encode5/batch_normalization_11/gamma_0',
    'encode5/batch_normalization_11/beta_0',
    'encode5/batch_normalization_11/moving_mean_0',
    'encode5/batch_normalization_11/moving_variance_0',
    'encode6/conv1d_12/kernel_0',
    'encode6/conv1d_12/bias_0',
    'encode6/batch_normalization_12/gamma_0',
    'encode6/batch_normalization_12/beta_0',
    'encode6/batch_normalization_12/moving_mean_0',
    'encode6/batch_normalization_12/moving_variance_0',
    'encode6/conv1d_13/kernel_0',
    'encode6/conv1d_13/bias_0',
    'encode6/batch_normalization_13/gamma_0',
    'encode6/batch_normalization_13/beta_0',
    'encode6/batch_normalization_13/moving_mean_0',
    'encode6/batch_normalization_13/moving_variance_0',
    'encode7/conv1d_14/kernel_0',
    'encode7/conv1d_14/bias_0',
    'encode7/batch_normalization_14/gamma_0',
    'encode7/batch_normalization_14/beta_0',
    'encode7/batch_normalization_14/moving_mean_0',
    'encode7/batch_normalization_14/moving_variance_0',
    'encode7/conv1d_15/kernel_0',
    'encode7/conv1d_15/bias_0',
    'encode7/batch_normalization_15/gamma_0',
    'encode7/batch_normalization_15/beta_0',
    'encode7/batch_normalization_15/moving_mean_0',
    'encode7/batch_normalization_15/moving_variance_0',
    'encode8/conv1d_16/kernel_0',
    'encode8/conv1d_16/bias_0',
    'encode8/batch_normalization_16/gamma_0',
    'encode8/batch_normalization_16/beta_0',
    'encode8/batch_normalization_16/moving_mean_0',
    'encode8/batch_normalization_16/moving_variance_0',
    'encode8/conv1d_17/kernel_0',
    'encode8/conv1d_17/bias_0',
    'encode8/batch_normalization_17/gamma_0',
    'encode8/batch_normalization_17/beta_0',
    'encode8/batch_normalization_17/moving_mean_0',
    'encode8/batch_normalization_17/moving_variance_0',
    'two_conv_center/conv1d_18/kernel_0',
    'two_conv_center/conv1d_18/bias_0',
    'two_conv_center/batch_normalization_18/gamma_0',
    'two_conv_center/batch_normalization_18/beta_0',
    'two_conv_center/batch_normalization_18/moving_mean_0',
    'two_conv_center/batch_normalization_18/moving_variance_0',
    'two_conv_center/conv1d_19/kernel_0',
    'two_conv_center/conv1d_19/bias_0',
    'two_conv_center/batch_normalization_19/gamma_0',
    'two_conv_center/batch_normalization_19/beta_0',
    'two_conv_center/batch_normalization_19/moving_mean_0',
    'two_conv_center/batch_normalization_19/moving_variance_0',
    'conv_transpose_decoder8/conv1d_transpose/kernel_0',
    'conv_transpose_decoder8/conv1d_transpose/bias_0',
    'conv_transpose_decoder8/batch_normalization_20/gamma_0',
    'conv_transpose_decoder8/batch_normalization_20/beta_0',
    'conv_transpose_decoder8/batch_normalization_20/moving_mean_0',
    'conv_transpose_decoder8/batch_normalization_20/moving_variance_0',
    'two_conv_decoder8/conv1d_20/kernel_0',
    'two_conv_decoder8/conv1d_20/bias_0',
    'two_conv_decoder8/batch_normalization_21/gamma_0',
    'two_conv_decoder8/batch_normalization_21/beta_0',
    'two_conv_decoder8/batch_normalization_21/moving_mean_0',
    'two_conv_decoder8/batch_normalization_21/moving_variance_0',
    'two_conv_decoder8/conv1d_21/kernel_0',
    'two_conv_decoder8/conv1d_21/bias_0',
    'two_conv_decoder8/batch_normalization_22/gamma_0',
    'two_conv_decoder8/batch_normalization_22/beta_0',
    'two_conv_decoder8/batch_normalization_22/moving_mean_0',
    'two_conv_decoder8/batch_normalization_22/moving_variance_0',
    'conv_transpose_decoder7/conv1d_transpose_1/kernel_0',
    'conv_transpose_decoder7/conv1d_transpose_1/bias_0',
    'conv_transpose_decoder7/batch_normalization_23/gamma_0',
    'conv_transpose_decoder7/batch_normalization_23/beta_0',
    'conv_transpose_decoder7/batch_normalization_23/moving_mean_0',
    'conv_transpose_decoder7/batch_normalization_23/moving_variance_0',
    'two_conv_decoder7/conv1d_22/kernel_0',
    'two_conv_decoder7/conv1d_22/bias_0',
    'two_conv_decoder7/batch_normalization_24/gamma_0',
    'two_conv_decoder7/batch_normalization_24/beta_0',
    'two_conv_decoder7/batch_normalization_24/moving_mean_0',
    'two_conv_decoder7/batch_normalization_24/moving_variance_0',
    'two_conv_decoder7/conv1d_23/kernel_0',
    'two_conv_decoder7/conv1d_23/bias_0',
    'two_conv_decoder7/batch_normalization_25/gamma_0',
    'two_conv_decoder7/batch_normalization_25/beta_0',
    'two_conv_decoder7/batch_normalization_25/moving_mean_0',
    'two_conv_decoder7/batch_normalization_25/moving_variance_0',
    'conv_transpose_decoder6/conv1d_transpose_2/kernel_0',
    'conv_transpose_decoder6/conv1d_transpose_2/bias_0',
    'conv_transpose_decoder6/batch_normalization_26/gamma_0',
    'conv_transpose_decoder6/batch_normalization_26/beta_0',
    'conv_transpose_decoder6/batch_normalization_26/moving_mean_0',
    'conv_transpose_decoder6/batch_normalization_26/moving_variance_0',
    'two_conv_decoder6/conv1d_24/kernel_0',
    'two_conv_decoder6/conv1d_24/bias_0',
    'two_conv_decoder6/batch_normalization_27/gamma_0',
    'two_conv_decoder6/batch_normalization_27/beta_0',
    'two_conv_decoder6/batch_normalization_27/moving_mean_0',
    'two_conv_decoder6/batch_normalization_27/moving_variance_0',
    'two_conv_decoder6/conv1d_25/kernel_0',
    'two_conv_decoder6/conv1d_25/bias_0',
    'two_conv_decoder6/batch_normalization_28/gamma_0',
    'two_conv_decoder6/batch_normalization_28/beta_0',
    'two_conv_decoder6/batch_normalization_28/moving_mean_0',
    'two_conv_decoder6/batch_normalization_28/moving_variance_0',
    'conv_transpose_decoder5/conv1d_transpose_3/kernel_0',
    'conv_transpose_decoder5/conv1d_transpose_3/bias_0',
    'conv_transpose_decoder5/batch_normalization_29/gamma_0',
    'conv_transpose_decoder5/batch_normalization_29/beta_0',
    'conv_transpose_decoder5/batch_normalization_29/moving_mean_0',
    'conv_transpose_decoder5/batch_normalization_29/moving_variance_0',
    'two_conv_decoder5/conv1d_26/kernel_0',
    'two_conv_decoder5/conv1d_26/bias_0',
    'two_conv_decoder5/batch_normalization_30/gamma_0',
    'two_conv_decoder5/batch_normalization_30/beta_0',
    'two_conv_decoder5/batch_normalization_30/moving_mean_0',
    'two_conv_decoder5/batch_normalization_30/moving_variance_0',
    'two_conv_decoder5/conv1d_27/kernel_0',
    'two_conv_decoder5/conv1d_27/bias_0',
    'two_conv_decoder5/batch_normalization_31/gamma_0',
    'two_conv_decoder5/batch_normalization_31/beta_0',
    'two_conv_decoder5/batch_normalization_31/moving_mean_0',
    'two_conv_decoder5/batch_normalization_31/moving_variance_0',
    'conv_transpose_decoder4/conv1d_transpose_4/kernel_0',
    'conv_transpose_decoder4/conv1d_transpose_4/bias_0',
    'conv_transpose_decoder4/batch_normalization_32/gamma_0',
    'conv_transpose_decoder4/batch_normalization_32/beta_0',
    'conv_transpose_decoder4/batch_normalization_32/moving_mean_0',
    'conv_transpose_decoder4/batch_normalization_32/moving_variance_0',
    'two_conv_decoder4/conv1d_28/kernel_0',
    'two_conv_decoder4/conv1d_28/bias_0',
    'two_conv_decoder4/batch_normalization_33/gamma_0',
    'two_conv_decoder4/batch_normalization_33/beta_0',
    'two_conv_decoder4/batch_normalization_33/moving_mean_0',
    'two_conv_decoder4/batch_normalization_33/moving_variance_0',
    'two_conv_decoder4/conv1d_29/kernel_0',
    'two_conv_decoder4/conv1d_29/bias_0',
    'two_conv_decoder4/batch_normalization_34/gamma_0',
    'two_conv_decoder4/batch_normalization_34/beta_0',
    'two_conv_decoder4/batch_normalization_34/moving_mean_0',
    'two_conv_decoder4/batch_normalization_34/moving_variance_0',
    'conv_transpose_decoder3/conv1d_transpose_5/kernel_0',
    'conv_transpose_decoder3/conv1d_transpose_5/bias_0',
    'conv_transpose_decoder3/batch_normalization_35/gamma_0',
    'conv_transpose_decoder3/batch_normalization_35/beta_0',
    'conv_transpose_decoder3/batch_normalization_35/moving_mean_0',
    'conv_transpose_decoder3/batch_normalization_35/moving_variance_0',
    'two_conv_decoder3/conv1d_30/kernel_0',
    'two_conv_decoder3/conv1d_30/bias_0',
    'two_conv_decoder3/batch_normalization_36/gamma_0',
    'two_conv_decoder3/batch_normalization_36/beta_0',
    'two_conv_decoder3/batch_normalization_36/moving_mean_0',
    'two_conv_decoder3/batch_normalization_36/moving_variance_0',
    'two_conv_decoder3/conv1d_31/kernel_0',
    'two_conv_decoder3/conv1d_31/bias_0',
    'two_conv_decoder3/batch_normalization_37/gamma_0',
    'two_conv_decoder3/batch_normalization_37/beta_0',
    'two_conv_decoder3/batch_normalization_37/moving_mean_0',
    'two_conv_decoder3/batch_normalization_37/moving_variance_0',
    'conv_transpose_decoder2/conv1d_transpose_6/kernel_0',
    'conv_transpose_decoder2/conv1d_transpose_6/bias_0',
    'conv_transpose_decoder2/batch_normalization_38/gamma_0',
    'conv_transpose_decoder2/batch_normalization_38/beta_0',
    'conv_transpose_decoder2/batch_normalization_38/moving_mean_0',
    'conv_transpose_decoder2/batch_normalization_38/moving_variance_0',
    'two_conv_decoder2/conv1d_32/kernel_0',
    'two_conv_decoder2/conv1d_32/bias_0',
    'two_conv_decoder2/batch_normalization_39/gamma_0',
    'two_conv_decoder2/batch_normalization_39/beta_0',
    'two_conv_decoder2/batch_normalization_39/moving_mean_0',
    'two_conv_decoder2/batch_normalization_39/moving_variance_0',
    'two_conv_decoder2/conv1d_33/kernel_0',
    'two_conv_decoder2/conv1d_33/bias_0',
    'two_conv_decoder2/batch_normalization_40/gamma_0',
    'two_conv_decoder2/batch_normalization_40/beta_0',
    'two_conv_decoder2/batch_normalization_40/moving_mean_0',
    'two_conv_decoder2/batch_normalization_40/moving_variance_0',
    'conv_transpose_decoder1/conv1d_transpose_7/kernel_0',
    'conv_transpose_decoder1/conv1d_transpose_7/bias_0',
    'conv_transpose_decoder1/batch_normalization_41/gamma_0',
    'conv_transpose_decoder1/batch_normalization_41/beta_0',
    'conv_transpose_decoder1/batch_normalization_41/moving_mean_0',
    'conv_transpose_decoder1/batch_normalization_41/moving_variance_0',
    'two_conv_decoder1/conv1d_34/kernel_0',
    'two_conv_decoder1/conv1d_34/bias_0',
    'two_conv_decoder1/batch_normalization_42/gamma_0',
    'two_conv_decoder1/batch_normalization_42/beta_0',
    'two_conv_decoder1/batch_normalization_42/moving_mean_0',
    'two_conv_decoder1/batch_normalization_42/moving_variance_0',
    'two_conv_decoder1/conv1d_35/kernel_0',
    'two_conv_decoder1/conv1d_35/bias_0',
    'two_conv_decoder1/batch_normalization_43/gamma_0',
    'two_conv_decoder1/batch_normalization_43/beta_0',
    'two_conv_decoder1/batch_normalization_43/moving_mean_0',
    'two_conv_decoder1/batch_normalization_43/moving_variance_0',
    'conv_transpose_decoder0/conv1d_transpose_8/kernel_0',
    'conv_transpose_decoder0/conv1d_transpose_8/bias_0',
    'conv_transpose_decoder0/batch_normalization_44/gamma_0',
    'conv_transpose_decoder0/batch_normalization_44/beta_0',
    'conv_transpose_decoder0/batch_normalization_44/moving_mean_0',
    'conv_transpose_decoder0/batch_normalization_44/moving_variance_0',
    'two_conv_decoder0/conv1d_36/kernel_0',
    'two_conv_decoder0/conv1d_36/bias_0',
    'two_conv_decoder0/batch_normalization_45/gamma_0',
    'two_conv_decoder0/batch_normalization_45/beta_0',
    'two_conv_decoder0/batch_normalization_45/moving_mean_0',
    'two_conv_decoder0/batch_normalization_45/moving_variance_0',
    'two_conv_decoder0/conv1d_37/kernel_0',
    'two_conv_decoder0/conv1d_37/bias_0',
    'two_conv_decoder0/batch_normalization_46/gamma_0',
    'two_conv_decoder0/batch_normalization_46/beta_0',
    'two_conv_decoder0/batch_normalization_46/moving_mean_0',
    'two_conv_decoder0/batch_normalization_46/moving_variance_0',
    'conv1d_38/kernel_0',
    'conv1d_38/bias_0'],
   'scalars': ['epoch_loss',
    'epoch_tp',
    'epoch_fp',
    'epoch_tn',
    'epoch_fn',
    'epoch_precision',
    'epoch_recall',
    'epoch_accuracy',
    'epoch_auc'],
   'distributions': ['encode0/conv1d/kernel_0',
    'encode0/conv1d/bias_0',
    'encode0/batch_normalization/gamma_0',
    'encode0/batch_normalization/beta_0',
    'encode0/batch_normalization/moving_mean_0',
    'encode0/batch_normalization/moving_variance_0',
    'encode0/conv1d_1/kernel_0',
    'encode0/conv1d_1/bias_0',
    'encode0/batch_normalization_1/gamma_0',
    'encode0/batch_normalization_1/beta_0',
    'encode0/batch_normalization_1/moving_mean_0',
    'encode0/batch_normalization_1/moving_variance_0',
    'encode1/conv1d_2/kernel_0',
    'encode1/conv1d_2/bias_0',
    'encode1/batch_normalization_2/gamma_0',
    'encode1/batch_normalization_2/beta_0',
    'encode1/batch_normalization_2/moving_mean_0',
    'encode1/batch_normalization_2/moving_variance_0',
    'encode1/conv1d_3/kernel_0',
    'encode1/conv1d_3/bias_0',
    'encode1/batch_normalization_3/gamma_0',
    'encode1/batch_normalization_3/beta_0',
    'encode1/batch_normalization_3/moving_mean_0',
    'encode1/batch_normalization_3/moving_variance_0',
    'encode2/conv1d_4/kernel_0',
    'encode2/conv1d_4/bias_0',
    'encode2/batch_normalization_4/gamma_0',
    'encode2/batch_normalization_4/beta_0',
    'encode2/batch_normalization_4/moving_mean_0',
    'encode2/batch_normalization_4/moving_variance_0',
    'encode2/conv1d_5/kernel_0',
    'encode2/conv1d_5/bias_0',
    'encode2/batch_normalization_5/gamma_0',
    'encode2/batch_normalization_5/beta_0',
    'encode2/batch_normalization_5/moving_mean_0',
    'encode2/batch_normalization_5/moving_variance_0',
    'encode3/conv1d_6/kernel_0',
    'encode3/conv1d_6/bias_0',
    'encode3/batch_normalization_6/gamma_0',
    'encode3/batch_normalization_6/beta_0',
    'encode3/batch_normalization_6/moving_mean_0',
    'encode3/batch_normalization_6/moving_variance_0',
    'encode3/conv1d_7/kernel_0',
    'encode3/conv1d_7/bias_0',
    'encode3/batch_normalization_7/gamma_0',
    'encode3/batch_normalization_7/beta_0',
    'encode3/batch_normalization_7/moving_mean_0',
    'encode3/batch_normalization_7/moving_variance_0',
    'encode4/conv1d_8/kernel_0',
    'encode4/conv1d_8/bias_0',
    'encode4/batch_normalization_8/gamma_0',
    'encode4/batch_normalization_8/beta_0',
    'encode4/batch_normalization_8/moving_mean_0',
    'encode4/batch_normalization_8/moving_variance_0',
    'encode4/conv1d_9/kernel_0',
    'encode4/conv1d_9/bias_0',
    'encode4/batch_normalization_9/gamma_0',
    'encode4/batch_normalization_9/beta_0',
    'encode4/batch_normalization_9/moving_mean_0',
    'encode4/batch_normalization_9/moving_variance_0',
    'encode5/conv1d_10/kernel_0',
    'encode5/conv1d_10/bias_0',
    'encode5/batch_normalization_10/gamma_0',
    'encode5/batch_normalization_10/beta_0',
    'encode5/batch_normalization_10/moving_mean_0',
    'encode5/batch_normalization_10/moving_variance_0',
    'encode5/conv1d_11/kernel_0',
    'encode5/conv1d_11/bias_0',
    'encode5/batch_normalization_11/gamma_0',
    'encode5/batch_normalization_11/beta_0',
    'encode5/batch_normalization_11/moving_mean_0',
    'encode5/batch_normalization_11/moving_variance_0',
    'encode6/conv1d_12/kernel_0',
    'encode6/conv1d_12/bias_0',
    'encode6/batch_normalization_12/gamma_0',
    'encode6/batch_normalization_12/beta_0',
    'encode6/batch_normalization_12/moving_mean_0',
    'encode6/batch_normalization_12/moving_variance_0',
    'encode6/conv1d_13/kernel_0',
    'encode6/conv1d_13/bias_0',
    'encode6/batch_normalization_13/gamma_0',
    'encode6/batch_normalization_13/beta_0',
    'encode6/batch_normalization_13/moving_mean_0',
    'encode6/batch_normalization_13/moving_variance_0',
    'encode7/conv1d_14/kernel_0',
    'encode7/conv1d_14/bias_0',
    'encode7/batch_normalization_14/gamma_0',
    'encode7/batch_normalization_14/beta_0',
    'encode7/batch_normalization_14/moving_mean_0',
    'encode7/batch_normalization_14/moving_variance_0',
    'encode7/conv1d_15/kernel_0',
    'encode7/conv1d_15/bias_0',
    'encode7/batch_normalization_15/gamma_0',
    'encode7/batch_normalization_15/beta_0',
    'encode7/batch_normalization_15/moving_mean_0',
    'encode7/batch_normalization_15/moving_variance_0',
    'encode8/conv1d_16/kernel_0',
    'encode8/conv1d_16/bias_0',
    'encode8/batch_normalization_16/gamma_0',
    'encode8/batch_normalization_16/beta_0',
    'encode8/batch_normalization_16/moving_mean_0',
    'encode8/batch_normalization_16/moving_variance_0',
    'encode8/conv1d_17/kernel_0',
    'encode8/conv1d_17/bias_0',
    'encode8/batch_normalization_17/gamma_0',
    'encode8/batch_normalization_17/beta_0',
    'encode8/batch_normalization_17/moving_mean_0',
    'encode8/batch_normalization_17/moving_variance_0',
    'two_conv_center/conv1d_18/kernel_0',
    'two_conv_center/conv1d_18/bias_0',
    'two_conv_center/batch_normalization_18/gamma_0',
    'two_conv_center/batch_normalization_18/beta_0',
    'two_conv_center/batch_normalization_18/moving_mean_0',
    'two_conv_center/batch_normalization_18/moving_variance_0',
    'two_conv_center/conv1d_19/kernel_0',
    'two_conv_center/conv1d_19/bias_0',
    'two_conv_center/batch_normalization_19/gamma_0',
    'two_conv_center/batch_normalization_19/beta_0',
    'two_conv_center/batch_normalization_19/moving_mean_0',
    'two_conv_center/batch_normalization_19/moving_variance_0',
    'conv_transpose_decoder8/conv1d_transpose/kernel_0',
    'conv_transpose_decoder8/conv1d_transpose/bias_0',
    'conv_transpose_decoder8/batch_normalization_20/gamma_0',
    'conv_transpose_decoder8/batch_normalization_20/beta_0',
    'conv_transpose_decoder8/batch_normalization_20/moving_mean_0',
    'conv_transpose_decoder8/batch_normalization_20/moving_variance_0',
    'two_conv_decoder8/conv1d_20/kernel_0',
    'two_conv_decoder8/conv1d_20/bias_0',
    'two_conv_decoder8/batch_normalization_21/gamma_0',
    'two_conv_decoder8/batch_normalization_21/beta_0',
    'two_conv_decoder8/batch_normalization_21/moving_mean_0',
    'two_conv_decoder8/batch_normalization_21/moving_variance_0',
    'two_conv_decoder8/conv1d_21/kernel_0',
    'two_conv_decoder8/conv1d_21/bias_0',
    'two_conv_decoder8/batch_normalization_22/gamma_0',
    'two_conv_decoder8/batch_normalization_22/beta_0',
    'two_conv_decoder8/batch_normalization_22/moving_mean_0',
    'two_conv_decoder8/batch_normalization_22/moving_variance_0',
    'conv_transpose_decoder7/conv1d_transpose_1/kernel_0',
    'conv_transpose_decoder7/conv1d_transpose_1/bias_0',
    'conv_transpose_decoder7/batch_normalization_23/gamma_0',
    'conv_transpose_decoder7/batch_normalization_23/beta_0',
    'conv_transpose_decoder7/batch_normalization_23/moving_mean_0',
    'conv_transpose_decoder7/batch_normalization_23/moving_variance_0',
    'two_conv_decoder7/conv1d_22/kernel_0',
    'two_conv_decoder7/conv1d_22/bias_0',
    'two_conv_decoder7/batch_normalization_24/gamma_0',
    'two_conv_decoder7/batch_normalization_24/beta_0',
    'two_conv_decoder7/batch_normalization_24/moving_mean_0',
    'two_conv_decoder7/batch_normalization_24/moving_variance_0',
    'two_conv_decoder7/conv1d_23/kernel_0',
    'two_conv_decoder7/conv1d_23/bias_0',
    'two_conv_decoder7/batch_normalization_25/gamma_0',
    'two_conv_decoder7/batch_normalization_25/beta_0',
    'two_conv_decoder7/batch_normalization_25/moving_mean_0',
    'two_conv_decoder7/batch_normalization_25/moving_variance_0',
    'conv_transpose_decoder6/conv1d_transpose_2/kernel_0',
    'conv_transpose_decoder6/conv1d_transpose_2/bias_0',
    'conv_transpose_decoder6/batch_normalization_26/gamma_0',
    'conv_transpose_decoder6/batch_normalization_26/beta_0',
    'conv_transpose_decoder6/batch_normalization_26/moving_mean_0',
    'conv_transpose_decoder6/batch_normalization_26/moving_variance_0',
    'two_conv_decoder6/conv1d_24/kernel_0',
    'two_conv_decoder6/conv1d_24/bias_0',
    'two_conv_decoder6/batch_normalization_27/gamma_0',
    'two_conv_decoder6/batch_normalization_27/beta_0',
    'two_conv_decoder6/batch_normalization_27/moving_mean_0',
    'two_conv_decoder6/batch_normalization_27/moving_variance_0',
    'two_conv_decoder6/conv1d_25/kernel_0',
    'two_conv_decoder6/conv1d_25/bias_0',
    'two_conv_decoder6/batch_normalization_28/gamma_0',
    'two_conv_decoder6/batch_normalization_28/beta_0',
    'two_conv_decoder6/batch_normalization_28/moving_mean_0',
    'two_conv_decoder6/batch_normalization_28/moving_variance_0',
    'conv_transpose_decoder5/conv1d_transpose_3/kernel_0',
    'conv_transpose_decoder5/conv1d_transpose_3/bias_0',
    'conv_transpose_decoder5/batch_normalization_29/gamma_0',
    'conv_transpose_decoder5/batch_normalization_29/beta_0',
    'conv_transpose_decoder5/batch_normalization_29/moving_mean_0',
    'conv_transpose_decoder5/batch_normalization_29/moving_variance_0',
    'two_conv_decoder5/conv1d_26/kernel_0',
    'two_conv_decoder5/conv1d_26/bias_0',
    'two_conv_decoder5/batch_normalization_30/gamma_0',
    'two_conv_decoder5/batch_normalization_30/beta_0',
    'two_conv_decoder5/batch_normalization_30/moving_mean_0',
    'two_conv_decoder5/batch_normalization_30/moving_variance_0',
    'two_conv_decoder5/conv1d_27/kernel_0',
    'two_conv_decoder5/conv1d_27/bias_0',
    'two_conv_decoder5/batch_normalization_31/gamma_0',
    'two_conv_decoder5/batch_normalization_31/beta_0',
    'two_conv_decoder5/batch_normalization_31/moving_mean_0',
    'two_conv_decoder5/batch_normalization_31/moving_variance_0',
    'conv_transpose_decoder4/conv1d_transpose_4/kernel_0',
    'conv_transpose_decoder4/conv1d_transpose_4/bias_0',
    'conv_transpose_decoder4/batch_normalization_32/gamma_0',
    'conv_transpose_decoder4/batch_normalization_32/beta_0',
    'conv_transpose_decoder4/batch_normalization_32/moving_mean_0',
    'conv_transpose_decoder4/batch_normalization_32/moving_variance_0',
    'two_conv_decoder4/conv1d_28/kernel_0',
    'two_conv_decoder4/conv1d_28/bias_0',
    'two_conv_decoder4/batch_normalization_33/gamma_0',
    'two_conv_decoder4/batch_normalization_33/beta_0',
    'two_conv_decoder4/batch_normalization_33/moving_mean_0',
    'two_conv_decoder4/batch_normalization_33/moving_variance_0',
    'two_conv_decoder4/conv1d_29/kernel_0',
    'two_conv_decoder4/conv1d_29/bias_0',
    'two_conv_decoder4/batch_normalization_34/gamma_0',
    'two_conv_decoder4/batch_normalization_34/beta_0',
    'two_conv_decoder4/batch_normalization_34/moving_mean_0',
    'two_conv_decoder4/batch_normalization_34/moving_variance_0',
    'conv_transpose_decoder3/conv1d_transpose_5/kernel_0',
    'conv_transpose_decoder3/conv1d_transpose_5/bias_0',
    'conv_transpose_decoder3/batch_normalization_35/gamma_0',
    'conv_transpose_decoder3/batch_normalization_35/beta_0',
    'conv_transpose_decoder3/batch_normalization_35/moving_mean_0',
    'conv_transpose_decoder3/batch_normalization_35/moving_variance_0',
    'two_conv_decoder3/conv1d_30/kernel_0',
    'two_conv_decoder3/conv1d_30/bias_0',
    'two_conv_decoder3/batch_normalization_36/gamma_0',
    'two_conv_decoder3/batch_normalization_36/beta_0',
    'two_conv_decoder3/batch_normalization_36/moving_mean_0',
    'two_conv_decoder3/batch_normalization_36/moving_variance_0',
    'two_conv_decoder3/conv1d_31/kernel_0',
    'two_conv_decoder3/conv1d_31/bias_0',
    'two_conv_decoder3/batch_normalization_37/gamma_0',
    'two_conv_decoder3/batch_normalization_37/beta_0',
    'two_conv_decoder3/batch_normalization_37/moving_mean_0',
    'two_conv_decoder3/batch_normalization_37/moving_variance_0',
    'conv_transpose_decoder2/conv1d_transpose_6/kernel_0',
    'conv_transpose_decoder2/conv1d_transpose_6/bias_0',
    'conv_transpose_decoder2/batch_normalization_38/gamma_0',
    'conv_transpose_decoder2/batch_normalization_38/beta_0',
    'conv_transpose_decoder2/batch_normalization_38/moving_mean_0',
    'conv_transpose_decoder2/batch_normalization_38/moving_variance_0',
    'two_conv_decoder2/conv1d_32/kernel_0',
    'two_conv_decoder2/conv1d_32/bias_0',
    'two_conv_decoder2/batch_normalization_39/gamma_0',
    'two_conv_decoder2/batch_normalization_39/beta_0',
    'two_conv_decoder2/batch_normalization_39/moving_mean_0',
    'two_conv_decoder2/batch_normalization_39/moving_variance_0',
    'two_conv_decoder2/conv1d_33/kernel_0',
    'two_conv_decoder2/conv1d_33/bias_0',
    'two_conv_decoder2/batch_normalization_40/gamma_0',
    'two_conv_decoder2/batch_normalization_40/beta_0',
    'two_conv_decoder2/batch_normalization_40/moving_mean_0',
    'two_conv_decoder2/batch_normalization_40/moving_variance_0',
    'conv_transpose_decoder1/conv1d_transpose_7/kernel_0',
    'conv_transpose_decoder1/conv1d_transpose_7/bias_0',
    'conv_transpose_decoder1/batch_normalization_41/gamma_0',
    'conv_transpose_decoder1/batch_normalization_41/beta_0',
    'conv_transpose_decoder1/batch_normalization_41/moving_mean_0',
    'conv_transpose_decoder1/batch_normalization_41/moving_variance_0',
    'two_conv_decoder1/conv1d_34/kernel_0',
    'two_conv_decoder1/conv1d_34/bias_0',
    'two_conv_decoder1/batch_normalization_42/gamma_0',
    'two_conv_decoder1/batch_normalization_42/beta_0',
    'two_conv_decoder1/batch_normalization_42/moving_mean_0',
    'two_conv_decoder1/batch_normalization_42/moving_variance_0',
    'two_conv_decoder1/conv1d_35/kernel_0',
    'two_conv_decoder1/conv1d_35/bias_0',
    'two_conv_decoder1/batch_normalization_43/gamma_0',
    'two_conv_decoder1/batch_normalization_43/beta_0',
    'two_conv_decoder1/batch_normalization_43/moving_mean_0',
    'two_conv_decoder1/batch_normalization_43/moving_variance_0',
    'conv_transpose_decoder0/conv1d_transpose_8/kernel_0',
    'conv_transpose_decoder0/conv1d_transpose_8/bias_0',
    'conv_transpose_decoder0/batch_normalization_44/gamma_0',
    'conv_transpose_decoder0/batch_normalization_44/beta_0',
    'conv_transpose_decoder0/batch_normalization_44/moving_mean_0',
    'conv_transpose_decoder0/batch_normalization_44/moving_variance_0',
    'two_conv_decoder0/conv1d_36/kernel_0',
    'two_conv_decoder0/conv1d_36/bias_0',
    'two_conv_decoder0/batch_normalization_45/gamma_0',
    'two_conv_decoder0/batch_normalization_45/beta_0',
    'two_conv_decoder0/batch_normalization_45/moving_mean_0',
    'two_conv_decoder0/batch_normalization_45/moving_variance_0',
    'two_conv_decoder0/conv1d_37/kernel_0',
    'two_conv_decoder0/conv1d_37/bias_0',
    'two_conv_decoder0/batch_normalization_46/gamma_0',
    'two_conv_decoder0/batch_normalization_46/beta_0',
    'two_conv_decoder0/batch_normalization_46/moving_mean_0',
    'two_conv_decoder0/batch_normalization_46/moving_variance_0',
    'conv1d_38/kernel_0',
    'conv1d_38/bias_0'],
   'tensors': ['batch_2'],
   'graph': False,
   'meta_graph': False,
   'run_metadata': []}

ea.Scalars('epoch_loss')

conda activate tensorflow_env
cd Programme/drmed-git
export MLFLOW_EXPERIMENT_NAME=exp-310520-unet
export MLFLOW_TRACKING_URI=file:./data/mlruns

mlflow experiments list

  Experiment Id  Name             Artifact Location
---------------  ---------------  --------------------
              0  exp-310520-unet  file:./data/mlruns/0
              1  exp-devtest      file:./data/mlruns/1

mlflow runs list --experiment-id 0

Date                      Name    ID
------------------------  ------  --------------------------------
2020-06-12 14:20:30 CEST          037e1d9e4ad74784974f4aaac11138cc
2020-05-31 21:39:51 CEST          1aefda1366f04f5da5d1fc2241ad9208

export EXP2=037e1d9e4ad74784974f4aaac11138cc

I have accidentally run the model with the wrong experiment ID. I'll check if I can change the experiment. I will do it later with mlflow.tracking.MlflowClient().rename_experiment()

mlflow artifacts list -r $EXP2
mlflow artifacts list -r $EXP2 -a model
mlflow artifacts list -r $EXP2 -a model_summary.txt
mlflow artifacts list -r $EXP2 -a tensorboard_logs/train/

(tensorflow_env) [lex@Topialex drmed-git]$ mlflow artifacts list -r $EXP2
[{
  "path": "model",
  "is_dir": true
}, {
  "path": "model_summary.txt",
  "is_dir": false,
  "file_size": "10895"
}, {
  "path": "tensorboard_logs",
  "is_dir": true
}]

(tensorflow_env) [lex@Topialex drmed-git]$ mlflow artifacts list -r $EXP2 -a model
[{
  "path": "model/MLmodel",
  "is_dir": false,
  "file_size": "317"
}, {
  "path": "model/conda.yaml",
  "is_dir": false,
  "file_size": "125"
}, {
  "path": "model/data",
  "is_dir": true
}]
(tensorflow_env) [lex@Topialex drmed-git]$ mlflow artifacts list -r $EXP2 -a model_summary.txt
[]

(tensorflow_env) [lex@Topialex drmed-git]$ mlflow artifacts list -r $EXP2 -a tensorboard_logs/train/
[{
  "path": "tensorboard_logs/train/events.out.tfevents.1591964813.node151.309868.9202.v2",
  "is_dir": false,
  "file_size": "20270485"
}, {
  "path": "tensorboard_logs/train/events.out.tfevents.1591964827.node151.profile-empty",
  "is_dir": false,
  "file_size": "40"
}, {
  "path": "tensorboard_logs/train/plugins",
  "is_dir": true
}]

mlflow artifacts download -r $EXP2
mlflow artifacts download -r $EXP2 -a model
mlflow artifacts download -r $EXP2 -a model_summary.txt
mlflow artifacts download -r $EXP2 -a tensorboard_logs

/home/lex/Programme/drmed-git/data/mlruns/0/037e1d9e4ad74784974f4aaac11138cc/artifacts
/home/lex/Programme/drmed-git/data/mlruns/0/037e1d9e4ad74784974f4aaac11138cc/artifacts/model
/home/lex/Programme/drmed-git/data/mlruns/0/037e1d9e4ad74784974f4aaac11138cc/artifacts/model_summary.txt
/home/lex/Programme/drmed-git/data/mlruns/0/037e1d9e4ad74784974f4aaac11138cc/artifacts/tensorboard_logs

mlflow runs describe --run-id $EXP2

{
    "info": {
        "artifact_uri": "file:./data/mlruns/0/037e1d9e4ad74784974f4aaac11138cc/artifacts",
        "end_time": 1592159296429,
        "experiment_id": "0",
        "lifecycle_stage": "active",
        "run_id": "037e1d9e4ad74784974f4aaac11138cc",
        "run_uuid": "037e1d9e4ad74784974f4aaac11138cc",
        "start_time": 1591964430233,
        "status": "FINISHED",
        "user_id": "ye53nis"
    },
    "data": {
        "metrics": {
            "learning rate": 1e-14,
            "val_loss": 0.3001052141189575,
            "precision": 0.9152822494506836,
            "fp": 2029354.0,
            "loss": 0.2597578763961792,
            "val_tp": 5185959.0,
            "recall": 0.9116151332855225,
            "accuracy": 0.9603750705718994,
            "val_precision": 0.9696021676063538,
            "val_recall": 0.8409230709075928,
            "lr": 1e-14,
            "fn": 2125714.0,
            "auc": 0.9819398522377014,
            "val_fp": 162584.0,
            "val_accuracy": 0.9563749432563782,
            "val_tn": 19884838.0,
            "val_auc": 0.9799740314483643,
            "val_fn": 981025.0,
            "tn": 78777592.0,
            "tp": 21924942.0
        },
        "params": {
            "opt_beta_1": "0.9",
            "validation_steps": "320",
            "opt_learning_rate": "0.001",
            "fluotracify_path": "/beegfs/ye53nis/drmed-git/src/",
            "opt_amsgrad": "False",
            "frac_val": "0.2",
            "batch_size": "5",
            "epochs": "100",
            "opt_decay": "0.0",
            "steps_per_epoch": "1280",
            "length_delimiter": "16384",
            "opt_name": "Adam",
            "opt_beta_2": "0.999",
            "csv_path": "/beegfs/ye53nis/saves/firstartefact_Sep2019",
            "learning_rate": "None",
            "opt_epsilon": "1e-07"
        },
        "tags": {
            "mlflow.source.git.repoURL": "https://github.com/aseltmann/fluotracify",
            "mlflow.source.type": "PROJECT",
            "mlflow.source.name": "file:///beegfs/ye53nis/drmed-git",
            "mlflow.user": "ye53nis",
            "mlflow.source.git.commit": "09c1d2a1bc083695026b46c74ea175c9168bb5f2",
            "mlflow.gitRepoURL": "https://github.com/aseltmann/fluotracify",
            "mlflow.project.backend": "local",
            "mlflow.project.env": "conda",
            "mlflow.project.entryPoint": "main",
            "mlflow.log-model.history": "[{
                \"run_id\": \"037e1d9e4ad74784974f4aaac11138cc\",
                \"artifact_path\": \"model\",
                \"utc_time_created\": \"2020-06-14 18:26:02.386982\",
                \"flavors\": {
                    \"keras\": {
                        \"keras_module\": \"tensorflow.keras\",
                        \"keras_version\": \"2.2.4-tf\",
                        \"data\": \"data\"
                    },
                    \"python_function\": {
                        \"loader_module\": \"mlflow.keras\",
                        \"python_version\": \"3.8.3\",
                        \"data\": \"data\",
                        \"env\": \"conda.yaml\"
                    }
                }
            }]"
        }
    }
}

tensorboard --logdir=data/mlruns/0/$EXP2/artifacts/tensorboard_logs

mlflow ui --backend-store-uri file:///home/lex/Programme/drmed-git/data/mlruns

AUC: Loss: Precision / Recall:

Prediction 03 Prediction 20 Prediction 24 Prediction 32 Prediction 99 Distribution of Conv1D-Kernel (final layer) Histograms of Conv1D-Kernel (final layer)

• more steps = good: better recall
• strong fluctuation of validation metrics in beginning and already convergence after ~30 Epochs for the second run → maybe use learning rate a bit more gently and start the lr schedule with lower rates
• judging from the tensorboard histograms, the model architecture can clearly be made simpler
• doing some background reading: in this SO thread it is argued that while using Adam, an extensive learning rate schedule, as I used, should not really be necessary.
  • Adam (adaptive moment estimation) updates any parameter with an individual learning rate
  • every learning rate can vary from 0 (no update) to the given learning rate as an upper limit
  • others argue, that it definitely helps (or that you have to start with a very low lr, which I can confirm from own experience) and that it especially could help reducing the loss in the late steps of training
  • finding optimal lr:
    • start with very very low lr, than increase till loss stops decreasing, and look at where the slope of the loss curve and pick the learning rate that is associated with the fastest decrease

1. Setup Tmux #+CALL: setup-tmux[:session local2]()

   rm:	cannot	remove	'home/lex.tmux-local-socket-remote-machine':	No	such	file	or	directory
   ye53nis@ara-login01.rz.uni-jena.de's	password:
   /tmp/tmux-67339/default
   >	ye53nis@ara-login01.rz.uni-jena.de's	password:

2. Request a compute node from Ara cluster

   cd /
   srun -p b_standard --time=7-10:00:00 --ntasks-per-node=24 --mem-per-cpu=2000 --pty bash
   

   (base) [ye53nis@node117 /]$
   

3. Make sure you are on the desired branch (exp-210104-unet)

   cd /beegfs/ye53nis/drmed-git
   git status
   

   (base) [ye53nis@node221 /]$ cd /beegfs/ye53nis/drmed-git
   (base) [ye53nis@node221 drmed-git]$ git status
   # On branch exp-210104-unet
   no changes added to commit (use "git add" and/or "git commit -a")
   (base) [ye53nis@node221 drmed-git]$
   

1. Setup Tmux (if we haven't done it before)

   sh-5.1$	ye53nis@ara-login01.rz.uni-jena.de's	password:
   >	ye53nis@ara-login01.rz.uni-jena.de's	password:

2. Request compute node from Ara cluster

   cd /
   srun -p s_standard --time=7-10:00:00 --ntasks-per-node=24 --mem-per-cpu=2000 --pty bash
   

3. Start Jupyter Lab

   (base) [ye53nis@login01 /]$ srun -p s_standard --time=7-10:00:00 --ntasks-per-node=24 --mem-per-cpu=2000 --pty bash
   (base) [ye53nis@node302 /]$ conda activate tf-nightly
   (tf-nightly) [ye53nis@node302 /]$ export PORT=9999
   (tf-nightly) [ye53nis@node302 /]$ export XDG_RUNTIME_DIR=''
   (tf-nightly) [ye53nis@node302 /]$ export XDG_RUNTIME_DIR=""
   (tf-nightly) [ye53nis@node302 /]$ jupyter lab --no-browser --port=$PORT
   [I 21:13:32.358 LabApp] JupyterLab extension loaded from /home/ye53nis/.conda/envs/tf-nightly/lib/python3.8/site-packages/jupyterlab
   [I 21:13:32.358 LabApp] JupyterLab application directory is /home/ye53nis/.conda/envs/tf-nightly/share/jupyter/lab
   [I 21:13:32.382 LabApp] Serving notebooks from local directory: /
   [I 21:13:32.382 LabApp] Jupyter Notebook 6.2.0 is running at:
   [I 21:13:32.383 LabApp] http://localhost:9999/?token=6f337c6b506799f0172382fc6a042886ebe3b00cd2a487e4
   [I 21:13:32.383 LabApp]  or http://127.0.0.1:9999/?token=6f337c6b506799f0172382fc6a042886ebe3b00cd2a487e4
   [I 21:13:32.383 LabApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
   [C 21:13:32.418 LabApp]
   
       To access the notebook, open this file in a browser:
           file:///home/ye53nis/.local/share/jupyter/runtime/nbserver-124120-open.html
       Or copy and paste one of these URLs:
           http://localhost:9999/?token=6f337c6b506799f0172382fc6a042886ebe3b00cd2a487e4
        or http://127.0.0.1:9999/?token=6f337c6b506799f0172382fc6a042886ebe3b00cd2a487e4
   

4. Tunnel the remote jupyter to the local computer

   sh-5.1$	sh-5.1$	ye53nis@ara-login01.rz.uni-jena.de's	password:
   ye53nis@node302's	password:
   Last	login:	Fri	Apr	2	19:25:33	2021	from	login01.ara

5. Start Python 3 kernel using jupyter-server-list-kernels. Then copy the kernel ID to the :PROPERTIES: drawer for any subtree where you want to use it

   python3           f26207a6-d326-45bb-b432-63d05a573ade   in a few seconds     starting   0
   

6. any files generated using the :file header of org-mode source blocks will be saved here

   (setq org-babel-jupyter-resource-directory "./data/exp-210204-unet")
   

   ./data/exp-210204-unet
   

conda activate tf-nightly
mlflow ui --backend-store-uri file:///beegfs/ye53nis/drmed-git/data/mlruns

(tf-nightly) [ye53nis@login01 data]$ mlflow ui --backend-store-uri file:///beegfs/ye53nis/drmed-git/data/mlruns
[2021-02-08 11:55:44 +0100] [19566] [INFO] Starting gunicorn 20.0.4
[2021-02-08 11:55:44 +0100] [19566] [INFO] Listening at: http://127.0.0.1:5000 (19566)
[2021-02-08 11:55:44 +0100] [19566] [INFO] Using worker: sync
[2021-02-08 11:55:44 +0100] [19572] [INFO] Booting worker with pid: 19572

1. current directory

   %cd /beegfs/ye53nis/drmed-git
   

   /beegfs/ye53nis/drmed-git
   

2. git log

   !git log -3
   

   commit bc590b22329ea0dad425a040bb23370b3e9de3d4
   Author: Apoplex <oligolex@vivaldi.net>
   Date:   Thu Feb 4 20:40:13 2021 +0100
   
       Increase readability in html export
   
   commit bb78ded17807e0ad259440f4ae4624d1576cbba5
   Author: Apoplex <oligolex@vivaldi.net>
   Date:   Mon Feb 1 21:44:15 2021 +0100
   
       Add changes from unet branch like unet prepro
   commit 06186e40f8d4d1aa786e9268651160e96ead2223
   Author: Apoplex <oligolex@vivaldi.net>
   Date:   Mon Feb 1 21:16:42 2021 +0100
   
       Add these updates to ptu_utils from unet code
   

3. Metadata of environment #+CALL: jp-metadata(_long='True)

   No of CPUs in system: 72
   No of CPUs the current process can use: 24
   load average: (16.0, 16.0, 16.05)
   os.uname():  posix.uname_result(sysname='Linux', nodename='node221', release='3.10.0-957.1.3.el7.x86_64', version='#1 SMP Thu Nov 29 14:49:43 UTC 2018', machine='x86_64')
   PID of process: 231630
   RAM total: 199G, RAM used: 21G, RAM free: 138G
   the current directory: /beegfs/ye53nis/drmed-git
   My disk usage:
   Filesystem           Size  Used Avail Use% Mounted on
   /dev/sda1             50G  3.2G   47G   7% /
   devtmpfs              94G     0   94G   0% /dev
   tmpfs                 94G  302M   94G   1% /dev/shm
   tmpfs                 94G  131M   94G   1% /run
   tmpfs                 94G     0   94G   0% /sys/fs/cgroup
   nfs03-ib:/pool/work  100T   78T   23T  78% /nfsdata
   nfs01-ib:/cluster    2.0T  412G  1.6T  21% /cluster
   nfs01-ib:/home        80T   64T   17T  80% /home
   nfs02-ib:/data01      88T   70T   19T  80% /data01
   /dev/sda6            169G   33M  169G   1% /local
   /dev/sda5            2.0G   34M  2.0G   2% /tmp
   /dev/sda3            6.0G  295M  5.8G   5% /var
   beegfs_nodev         524T  277T  248T  53% /beegfs
   tmpfs                 19G     0   19G   0% /run/user/67339# packages in environment at /home/ye53nis/.conda/envs/tf-nightly:
   #
   # Name                    Version                   Build  Channel
   _libgcc_mutex             0.1                        main
   absl-py                   0.11.0                   pypi_0    pypi
   alembic                   1.4.1                      py_0    conda-forge/label/main
   appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge/label/main
   argon2-cffi               20.1.0           py38h7b6447c_1
   asn1crypto                1.4.0              pyh9f0ad1d_0    conda-forge/label/main
   asteval                   0.9.16             pyh5ca1d4c_0    conda-forge/label/main
   astunparse                1.6.3                    pypi_0    pypi
   async_generator           1.10               pyhd3eb1b0_0
   attrs                     20.3.0             pyhd3eb1b0_0
   azure-core                1.10.0             pyhd8ed1ab_0    conda-forge/label/main
   azure-storage-blob        12.7.1             pyh44b312d_0    conda-forge/label/main
   backcall                  0.2.0              pyhd3eb1b0_0
   blas                      1.0                         mkl
   bleach                    3.2.3              pyhd3eb1b0_0
   blinker                   1.4                        py_1    conda-forge/label/main
   blosc                     1.20.1               hd408876_0
   brotli                    1.0.9                he6710b0_2
   brotlipy                  0.7.0           py38h27cfd23_1003
   brunsli                   0.1                  h2531618_0
   bzip2                     1.0.8                h7b6447c_0
   ca-certificates           2020.12.5            ha878542_0    conda-forge/label/main
   cachetools                4.2.1                    pypi_0    pypi
   certifi                   2020.12.5        py38h578d9bd_1    conda-forge/label/main
   cffi                      1.14.4           py38h261ae71_0
   chardet                   4.0.0           py38h06a4308_1003
   charls                    2.1.0                he6710b0_2
   click                     7.1.2              pyh9f0ad1d_0    conda-forge/label/main
   cloudpickle               1.6.0                      py_0    conda-forge/label/main
   configparser              5.0.1                      py_0    conda-forge/label/main
   cryptography              3.3.1            py38h3c74f83_0
   cycler                    0.10.0                   py38_0
   databricks-cli            0.9.1                      py_0    conda-forge/label/main
   dbus                      1.13.18              hb2f20db_0
   decorator                 4.4.2              pyhd3eb1b0_0
   defusedxml                0.6.0                      py_0
   docker-py                 4.4.1            py38h578d9bd_1    conda-forge/label/main
   docker-pycreds            0.4.0                      py_0    conda-forge/label/main
   entrypoints               0.3                      py38_0
   expat                     2.2.10               he6710b0_2
   fcsfiles                  2020.9.18                pypi_0    pypi
   flask                     1.1.2              pyh9f0ad1d_0    conda-forge/label/main
   flatbuffers               1.12                     pypi_0    pypi
   fontconfig                2.13.0               h9420a91_0
   freetype                  2.10.4               h5ab3b9f_0
   future                    0.18.2           py38h578d9bd_3    conda-forge/label/main
   gast                      0.4.0                    pypi_0    pypi
   giflib                    5.1.4                h14c3975_1
   gitdb                     4.0.5                      py_0    conda-forge/label/main
   gitpython                 3.1.12             pyhd8ed1ab_0    conda-forge/label/main
   glib                      2.66.1               h92f7085_0
   google-auth               1.24.0                   pypi_0    pypi
   google-auth-oauthlib      0.4.2                    pypi_0    pypi
   google-pasta              0.2.0                    pypi_0    pypi
   gorilla                   0.3.0                      py_0    conda-forge/label/main
   grpcio                    1.34.1                   pypi_0    pypi
   gst-plugins-base          1.14.0               h8213a91_2
   gstreamer                 1.14.0               h28cd5cc_2
   gunicorn                  20.0.4           py38h578d9bd_3    conda-forge/label/main
   h5py                      3.1.0                    pypi_0    pypi
   icu                       58.2                 he6710b0_3
   idna                      2.10               pyhd3eb1b0_0
   imagecodecs               2021.1.11        py38h581e88b_1
   importlib-metadata        2.0.0                      py_1
   importlib_metadata        2.0.0                         1
   intel-openmp              2020.2                      254
   ipykernel                 5.3.4            py38h5ca1d4c_0
   ipython                   7.19.0           py38hb070fc8_1
   ipython_genutils          0.2.0              pyhd3eb1b0_1
   isodate                   0.6.0                      py_1    conda-forge/label/main
   itsdangerous              1.1.0                      py_0    conda-forge/label/main
   jedi                      0.17.0                   py38_0
   jinja2                    2.11.2             pyhd3eb1b0_0
   jpeg                      9b                   h024ee3a_2
   json5                     0.9.5                      py_0
   jsonschema                3.2.0                      py_2
   jupyter_client            6.1.7                      py_0
   jupyter_core              4.7.0            py38h06a4308_0
   jupyterlab                2.2.6                      py_0
   jupyterlab_pygments       0.1.2                      py_0
   jupyterlab_server         1.2.0                      py_0
   jxrlib                    1.1                  h7b6447c_2
   keras-preprocessing       1.1.2                    pypi_0    pypi
   kiwisolver                1.3.0            py38h2531618_0
   lcms2                     2.11                 h396b838_0
   ld_impl_linux-64          2.33.1               h53a641e_7
   lerc                      2.2.1                h2531618_0
   libaec                    1.0.4                he6710b0_1
   libdeflate                1.7                  h27cfd23_5
   libedit                   3.1.20191231         h14c3975_1
   libffi                    3.3                  he6710b0_2
   libgcc-ng                 9.1.0                hdf63c60_0
   libgfortran-ng            7.3.0                hdf63c60_0
   libpng                    1.6.37               hbc83047_0
   libprotobuf               3.13.0.1             h8b12597_0    conda-forge/label/main
   libsodium                 1.0.18               h7b6447c_0
   libstdcxx-ng              9.1.0                hdf63c60_0
   libtiff                   4.1.0                h2733197_1
   libuuid                   1.0.3                h1bed415_2
   libwebp                   1.0.1                h8e7db2f_0
   libxcb                    1.14                 h7b6447c_0
   libxml2                   2.9.10               hb55368b_3
   libzopfli                 1.0.3                he6710b0_0
   lmfit                     1.0.1                      py_1    conda-forge/label/main
   lz4-c                     1.9.3                h2531618_0
   mako                      1.1.4              pyh44b312d_0    conda-forge/label/main
   markdown                  3.3.3                    pypi_0    pypi
   markupsafe                1.1.1            py38h7b6447c_0
   matplotlib                3.3.2                h06a4308_0
   matplotlib-base           3.3.2            py38h817c723_0
   mistune                   0.8.4           py38h7b6447c_1000
   mkl                       2020.2                      256
   mkl-service               2.3.0            py38he904b0f_0
   mkl_fft                   1.2.0            py38h23d657b_0
   mkl_random                1.1.1            py38h0573a6f_0
   mlflow                    1.13.1           py38h578d9bd_2    conda-forge/label/main
   msrest                    0.6.21             pyh44b312d_0    conda-forge/label/main
   multipletau               0.3.3                    pypi_0    pypi
   nbclient                  0.5.1                      py_0
   nbconvert                 6.0.7                    py38_0
   nbformat                  5.1.2              pyhd3eb1b0_1
   ncurses                   6.2                  he6710b0_1
   nest-asyncio              1.4.3              pyhd3eb1b0_0
   notebook                  6.2.0            py38h06a4308_0
   numpy                     1.19.2           py38h54aff64_0
   numpy-base                1.19.2           py38hfa32c7d_0
   oauthlib                  3.0.1                      py_0    conda-forge/label/main
   olefile                   0.46                       py_0
   openjpeg                  2.3.0                h05c96fa_1
   openssl                   1.1.1i               h27cfd23_0
   opt-einsum                3.3.0                    pypi_0    pypi
   packaging                 20.9               pyhd3eb1b0_0
   pandas                    1.2.1            py38ha9443f7_0
   pandoc                    2.11                 hb0f4dca_0
   pandocfilters             1.4.3            py38h06a4308_1
   parso                     0.8.1              pyhd3eb1b0_0
   pcre                      8.44                 he6710b0_0
   pexpect                   4.8.0              pyhd3eb1b0_3
   pickleshare               0.7.5           pyhd3eb1b0_1003
   pillow                    8.1.0            py38he98fc37_0
   pip                       20.3.3           py38h06a4308_0
   prometheus_client         0.9.0              pyhd3eb1b0_0
   prometheus_flask_exporter 0.18.1             pyh9f0ad1d_0    conda-forge/label/main
   prompt-toolkit            3.0.8                      py_0
   protobuf                  3.13.0.1         py38hadf7658_1    conda-forge/label/main
   ptyprocess                0.7.0              pyhd3eb1b0_2
   pyasn1                    0.4.8                    pypi_0    pypi
   pyasn1-modules            0.2.8                    pypi_0    pypi
   pycparser                 2.20                       py_2
   pygments                  2.7.4              pyhd3eb1b0_0
   pyjwt                     2.0.1              pyhd8ed1ab_0    conda-forge/label/main
   pyopenssl                 20.0.1             pyhd3eb1b0_1
   pyparsing                 2.4.7              pyhd3eb1b0_0
   pyqt                      5.9.2            py38h05f1152_4
   pyrsistent                0.17.3           py38h7b6447c_0
   pysocks                   1.7.1            py38h06a4308_0
   python                    3.8.5                h7579374_1
   python-dateutil           2.8.1              pyhd3eb1b0_0
   python-editor             1.0.4                      py_0    conda-forge/label/main
   python_abi                3.8                      1_cp38    conda-forge/label/main
   pytz                      2020.5             pyhd3eb1b0_0
   pyyaml                    5.3.1            py38h8df0ef7_1    conda-forge/label/main
   pyzmq                     20.0.0           py38h2531618_1
   qt                        5.9.7                h5867ecd_1
   querystring_parser        1.2.4                      py_0    conda-forge/label/main
   readline                  8.1                  h27cfd23_0
   requests                  2.25.1             pyhd3eb1b0_0
   requests-oauthlib         1.3.0              pyh9f0ad1d_0    conda-forge/label/main
   rsa                       4.7                      pypi_0    pypi
   scipy                     1.5.2            py38h0b6359f_0
   seaborn                   0.11.1             pyhd3eb1b0_0
   send2trash                1.5.0              pyhd3eb1b0_1
   setuptools                52.0.0           py38h06a4308_0
   sip                       4.19.13          py38he6710b0_0
   six                       1.15.0           py38h06a4308_0
   smmap                     4.0.0              pyh44b312d_0    conda-forge/label/main
   snappy                    1.1.8                he6710b0_0
   sqlalchemy                1.3.20           py38h1e0a361_0    conda-forge/label/main
   sqlite                    3.33.0               h62c20be_0
   sqlparse                  0.4.1              pyh9f0ad1d_0    conda-forge/label/main
   tabulate                  0.8.7              pyh9f0ad1d_0    conda-forge/label/main
   tb-nightly                2.5.0a20210130           pypi_0    pypi
   tensorboard-plugin-wit    1.8.0                    pypi_0    pypi
   termcolor                 1.1.0                    pypi_0    pypi
   terminado                 0.9.2            py38h06a4308_0
   testpath                  0.4.4              pyhd3eb1b0_0
   tf-estimator-nightly      2.5.0.dev2021020101          pypi_0    pypi
   tf-nightly                2.5.0.dev20210130          pypi_0    pypi
   tifffile                  2021.1.14          pyhd3eb1b0_1
   tk                        8.6.10               hbc83047_0
   tornado                   6.1              py38h27cfd23_0
   traitlets                 5.0.5              pyhd3eb1b0_0
   typing-extensions         3.7.4.3                  pypi_0    pypi
   uncertainties             3.1.5              pyhd8ed1ab_0    conda-forge/label/main
   urllib3                   1.26.3             pyhd3eb1b0_0
   wcwidth                   0.2.5                      py_0
   webencodings              0.5.1                    py38_1
   websocket-client          0.57.0           py38h578d9bd_4    conda-forge/label/main
   werkzeug                  1.0.1              pyh9f0ad1d_0    conda-forge/label/main
   wheel                     0.36.2             pyhd3eb1b0_0
   wrapt                     1.12.1                   pypi_0    pypi
   xz                        5.2.5                h7b6447c_0
   yaml                      0.2.5                h516909a_0    conda-forge/label/main
   zeromq                    4.3.3                he6710b0_3
   zfp                       0.5.5                h2531618_4
   zipp                      3.4.0              pyhd3eb1b0_0
   zlib                      1.2.11               h7b6447c_3
   zstd                      1.4.5                h9ceee32_0
   
   Note: you may need to restart the kernel to use updated packages.
   {'SLURM_CHECKPOINT_IMAGE_DIR': '/var/slurm/checkpoint',
    'SLURM_NODELIST': 'node221',
    'SLURM_JOB_NAME': 'bash',
    'XDG_SESSION_ID': '595',
    'SLURMD_NODENAME': 'node221',
    'SLURM_TOPOLOGY_ADDR': 'node221',
    'SLURM_NTASKS_PER_NODE': '24',
    'HOSTNAME': 'login01',
    'SLURM_PRIO_PROCESS': '0',
    'SLURM_SRUN_COMM_PORT': '33255',
    'SHELL': '/bin/bash',
    'TERM': 'xterm-color',
    'SLURM_JOB_QOS': 'qstand',
    'SLURM_PTY_WIN_ROW': '24',
    'HISTSIZE': '1000',
    'TMPDIR': '/tmp',
    'SLURM_TOPOLOGY_ADDR_PATTERN': 'node',
    'SSH_CLIENT': '10.231.182.213 44428 22',
    'CONDA_SHLVL': '2',
    'CONDA_PROMPT_MODIFIER': '(tf-nightly) ',
    'QTDIR': '/usr/lib64/qt-3.3',
    'QTINC': '/usr/lib64/qt-3.3/include',
    'SSH_TTY': '/dev/pts/2',
    'NO_PROXY': 'localhost,127.0.0.0/8,.uni-jena.de,141.35.0.0/16,10.0.0.0/8,192.168.0.0/16,172.0.0.0/8,fe80::/7,2001:638:1558::/24',
    'QT_GRAPHICSSYSTEM_CHECKED': '1',
    'SLURM_NNODES': '1',
    'USER': 'ye53nis',
    'http_proxy': 'http://internet4nzm.rz.uni-jena.de:3128',
    'LS_COLORS': 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:',
    'CONDA_EXE': '/cluster/miniconda3/bin/conda',
    'SLURM_STEP_NUM_NODES': '1',
    'SLURM_JOBID': '618588',
    'SRUN_DEBUG': '3',
    'FTP_PROXY': 'http://internet4nzm.rz.uni-jena.de:3128',
    'ftp_proxy': 'http://internet4nzm.rz.uni-jena.de:3128',
    'SLURM_NTASKS': '24',
    'SLURM_LAUNCH_NODE_IPADDR': '192.168.192.5',
    'SLURM_STEP_ID': '0',
    'TMUX': '/tmp/tmux-67339/default,43792,2',
    '_CE_CONDA': '',
    'CONDA_PREFIX_1': '/cluster/miniconda3',
    'SLURM_STEP_LAUNCHER_PORT': '33255',
    'SLURM_TASKS_PER_NODE': '24',
    'MAIL': '/var/spool/mail/ye53nis',
    'PATH': '/home/ye53nis/.conda/envs/tf-nightly/bin:/home/lex/Programme/miniconda3/envs/tf-nightly-lab/bin:/home/lex/Programme/miniconda3/condabin:/home/lex/.local/bin:/bin:/usr/bin:/usr/local/bin:/usr/local/sbin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/var/lib/snapd/snap/bin:/usr/sbin:/home/ye53nis/.local/bin:/home/ye53nis/bin',
    'SLURM_WORKING_CLUSTER': 'hpc:192.168.192.1:6817:8448',
    'SLURM_JOB_ID': '618588',
    'CONDA_PREFIX': '/home/ye53nis/.conda/envs/tf-nightly',
    'SLURM_JOB_USER': 'ye53nis',
    'SLURM_STEPID': '0',
    'PWD': '/',
    'SLURM_SRUN_COMM_HOST': '192.168.192.5',
    'LANG': 'en_US.UTF-8',
    'SLURM_PTY_WIN_COL': '80',
    'SLURM_UMASK': '0022',
    'MODULEPATH': '/usr/share/Modules/modulefiles:/etc/modulefiles:/cluster/modulefiles',
    'SLURM_JOB_UID': '67339',
    'LOADEDMODULES': '',
    'SLURM_NODEID': '0',
    'TMUX_PANE': '%2',
    'SLURM_SUBMIT_DIR': '/',
    'SLURM_TASK_PID': '75179',
    'SLURM_NPROCS': '24',
    'SLURM_CPUS_ON_NODE': '24',
    'SLURM_DISTRIBUTION': 'block',
    'HTTPS_PROXY': 'http://internet4nzm.rz.uni-jena.de:3128',
    'https_proxy': 'http://internet4nzm.rz.uni-jena.de:3128',
    'SLURM_PROCID': '0',
    'HISTCONTROL': 'ignoredups',
    '_CE_M': '',
    'SLURM_JOB_NODELIST': 'node221',
    'SLURM_PTY_PORT': '41588',
    'HOME': '/home/ye53nis',
    'SHLVL': '3',
    'SLURM_LOCALID': '0',
    'SLURM_JOB_GID': '13280',
    'SLURM_JOB_CPUS_PER_NODE': '24',
    'SLURM_CLUSTER_NAME': 'hpc',
    'no_proxy': 'localhost,127.0.0.0/8,.uni-jena.de,141.35.0.0/16,10.0.0.0/8,192.168.0.0/16,172.0.0.0/8,fe80::/7,2001:638:1558::/24',
    'SLURM_GTIDS': '0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23',
    'SLURM_SUBMIT_HOST': 'login01',
    'HTTP_PROXY': 'http://internet4nzm.rz.uni-jena.de:3128',
    'SLURM_JOB_PARTITION': 's_standard',
    'MATHEMATICA_HOME': '/cluster/apps/mathematica/11.3',
    'CONDA_PYTHON_EXE': '/cluster/miniconda3/bin/python',
    'LOGNAME': 'ye53nis',
    'SLURM_STEP_NUM_TASKS': '24',
    'QTLIB': '/usr/lib64/qt-3.3/lib',
    'SLURM_JOB_ACCOUNT': 'iaob',
    'SLURM_JOB_NUM_NODES': '1',
    'MODULESHOME': '/usr/share/Modules',
    'CONDA_DEFAULT_ENV': 'tf-nightly',
    'LESSOPEN': '||/usr/bin/lesspipe.sh %s',
    'SLURM_STEP_TASKS_PER_NODE': '24',
    'PORT': '9999','SLURM_STEP_NODELIST': 'node221',
   'DISPLAY': ':0',
   'XDG_RUNTIME_DIR': '',
   'XAUTHORITY': '/home/lex/.Xauthority',
   'BASH_FUNC_module()': '() {  eval `/usr/bin/modulecmd bash $*`\n}',
   '_': '/home/ye53nis/.conda/envs/tf-nightly/bin/jupyter',
   'JPY_PARENT_PID': '94022',
   'CLICOLOR': '1',
   'PAGER': 'cat',
   'GIT_PAGER': 'cat',
   'MPLBACKEND': 'module://ipykernel.pylab.backend_inline'}
   

• mlflow environment variables

  conda activate tf-nightly
  cd /beegfs/ye53nis/drmed-git
  export MLFLOW_EXPERIMENT_NAME=exp-210204-unet
  export MLFLOW_TRACKING_URI=file:./data/mlruns
  mkdir data/exp-210204-unet
  

  (tf-nightly) [ye53nis@node221 drmed-git]$
  

• Use whole dataset (6400 training, 1600 validation, 2000 test), but during training, use only 1/5th of it per epoch

  mlflow run . -P fluotracify_path=/beegfs/ye53nis/drmed-git/src/ -P epochs=50 -P learning_rate=None -P csv_path=/beegfs/ye53nis/saves/firstartifact_Nov2020 -P steps_per_epoch=1280 -P validation_steps=320
  

  (tf-nightly) [ye53nis@node221 drmed-git]$ mlflow run . -P fluotracify_path=/beegfs/ye53nis/drmed-git/src/ -P epochs=50 -P learning_rate=None -P csv_path=/beegfs/ye53nis/saves/firstartifact_Nov2020 -P steps_per_epoch=1280 -P validation_
  steps=320
  
  INFO: 'exp-210104-unet' does not exist. Creating a new experiment
  
  2021/02/05 18:23:53 INFO mlflow.projects.utils: === Created directory /tmp/tmptojz254v for downloading remote URIs passed to arguments of type 'path' ===
  
  2021/02/05 18:23:53 INFO mlflow.projects.backend.local:
  === Running command 'source /cluster/miniconda3/bin/../etc/profile.d/conda.sh && conda activate mlflow-fd9a200e5a24d4c79a0ff13be73ccb5141ed072c 1>&2
  && python src/fluotracify/training/train.py /beegfs/ye53nis/drmed-git/src 5 0.2 16384 None 50 /beegfs/ye53nis/saves/firstartifact_Nov2020 3 1280 320'
  in run with ID 'b9935d1e554c423fb2852242f4c4504c' ===
  
  2021-02-05 18:24:11.505650: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
  2021-02-05 18:24:11.505734: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. 2.5.0-dev20210130
  2021-02-05 18:24:48.480544: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
  2021-02-05 18:24:48.480629: W tensorflow/stream_executor/cuda/cuda_driver.cc:326] failed call to cuInit: UNKNOWN ERROR (303)
  2021-02-05 18:24:48.480678: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node221): /proc/driver/nvidia/version does not exist
  GPUs:  []
  
  train 0 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.1/traces_brightclust_Nov2020_D0.1_set007.csv
  train 1 /beegfs/ye53nis/saves/firstartifact_Nov2020/1.0/traces_brightclust_Nov2020_D1.0_set002.csv
  train 2 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.069/traces_brightclust_Nov2020_D0.069_set003.csv
  train 3 /beegfs/ye53nis/saves/firstartifact_Nov2020/1.0/traces_brightclust_Nov2020_D1.0_set003.csv
  train 4 /beegfs/ye53nis/saves/firstartifact_Nov2020/3.0/traces_brightclust_Nov2020_D3.0_set006.csv
  train 5 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.4/traces_brightclust_Nov2020_D0.4_set009.csv
  train 6 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.08/traces_brightclust_Nov2020_D0.08_set007.csv
  train 7 /beegfs/ye53nis/saves/firstartifact_Nov2020/50/traces_brightclust_Nov2020_D50_set010.csv
  train 8 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.6/traces_brightclust_Nov2020_D0.6_set007.csv
  train 9 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.4/traces_brightclust_Nov2020_D0.4_set006.csv
  train 10 /beegfs/ye53nis/saves/firstartifact_Nov2020/1.0/traces_brightclust_Nov2020_D1.0_set007.csv
  train 11 /beegfs/ye53nis/saves/firstartifact_Nov2020/50/traces_brightclust_Nov2020_D50_set007.csv
  train 12 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.6/traces_brightclust_Nov2020_D0.6_set002.csv
  train 13 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.08/traces_brightclust_Nov2020_D0.08_set004.csv
  train 14 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.069/traces_brightclust_Nov2020_D0.069_set008.csv
  train 15 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.2/traces_brightclust_Nov2020_D0.2_set001.csv
  train 16 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.1/traces_brightclust_Nov2020_D0.1_set003.csv
  train 17 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.1/traces_brightclust_Nov2020_D0.1_set005.csv
  train 18 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.2/traces_brightclust_Nov2020_D0.2_set004.csv
  train 19 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.069/traces_brightclust_Nov2020_D0.069_set009.csv
  train 20 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.4/traces_brightclust_Nov2020_D0.4_set004.csv
  train 21 /beegfs/ye53nis/saves/firstartifact_Nov2020/1.0/traces_brightclust_Nov2020_D1.0_set009.csv
  train 22 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.069/traces_brightclust_Nov2020_D0.069_set004.csv
  train 23 /beegfs/ye53nis/saves/firstartifact_Nov2020/10/traces_brightclust_Nov2020_D10_set005.csv
  train 24 /beegfs/ye53nis/saves/firstartifact_Nov2020/3.0/traces_brightclust_Nov2020_D3.0_set003.csv
  train 25 /beegfs/ye53nis/saves/firstartifact_Nov2020/10/traces_brightclust_Nov2020_D10_set006.csv
  train 26 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.069/traces_brightclust_Nov2020_D0.069_set007.csv
  train 27 /beegfs/ye53nis/saves/firstartifact_Nov2020/3.0/traces_brightclust_Nov2020_D3.0_set007.csv
  train 28 /beegfs/ye53nis/saves/firstartifact_Nov2020/3.0/traces_brightclust_Nov2020_D3.0_set005.csv
  train 29 /beegfs/ye53nis/saves/firstartifact_Nov2020/10/traces_brightclust_Nov2020_D10_set010.csv
  train 30 /beegfs/ye53nis/saves/firstartifact_Nov2020/1.0/traces_brightclust_Nov2020_D1.0_set004.csv
  train 31 /beegfs/ye53nis/saves/firstartifact_Nov2020/50/traces_brightclust_Nov2020_D50_set008.csv
  train 32 /beegfs/ye53nis/saves/firstartifact_Nov2020/50/traces_brightclust_Nov2020_D50_set001.csv
  train 33 /beegfs/ye53nis/saves/firstartifact_Nov2020/10/traces_brightclust_Nov2020_D10_set002.csv
  train 34 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.6/traces_brightclust_Nov2020_D0.6_set004.csv
  train 35 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.1/traces_brightclust_Nov2020_D0.1_set008.csv
  train 36 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.08/traces_brightclust_Nov2020_D0.08_set009.csv
  train 37 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.6/traces_brightclust_Nov2020_D0.6_set010.csv
  train 38 /beegfs/ye53nis/saves/firstartifact_Nov2020/3.0/traces_brightclust_Nov2020_D3.0_set010.csv
  train 39 /beegfs/ye53nis/saves/firstartifact_Nov2020/1.0/traces_brightclust_Nov2020_D1.0_set001.csv
  train 40 /beegfs/ye53nis/saves/firstartifact_Nov2020/50/traces_brightclust_Nov2020_D50_set004.csv
  train 41 /beegfs/ye53nis/saves/firstartifact_Nov2020/subsample/traces_brightclust_Nov2020_D3.0_set004.csv
  train 42 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.069/traces_brightclust_Nov2020_D0.069_set002.csv
  train 43 /beegfs/ye53nis/saves/firstartifact_Nov2020/10/traces_brightclust_Nov2020_D10_set007.csv
  train 44 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.4/traces_brightclust_Nov2020_D0.4_set003.csv
  train 45 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.4/traces_brightclust_Nov2020_D0.4_set002.csv
  train 46 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.069/traces_brightclust_Nov2020_D0.069_set005.csv
  train 47 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.08/traces_brightclust_Nov2020_D0.08_set006.csv
  train 48 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.08/traces_brightclust_Nov2020_D0.08_set008.csv
  train 49 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.4/traces_brightclust_Nov2020_D0.4_set001.csv
  train 50 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.2/traces_brightclust_Nov2020_D0.2_set009.csv
  train 51 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.069/traces_brightclust_Nov2020_D0.069_set006.csv
  train 52 /beegfs/ye53nis/saves/firstartifact_Nov2020/50/traces_brightclust_Nov2020_D50_set006.csv
  train 53 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.6/traces_brightclust_Nov2020_D0.6_set005.csv
  train 54 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.069/traces_brightclust_Nov2020_D0.069_set001.csv
  train 55 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.2/traces_brightclust_Nov2020_D0.2_set005.csv
  train 56 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.1/traces_brightclust_Nov2020_D0.1_set009.csv
  train 57 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.6/traces_brightclust_Nov2020_D0.6_set006.csv
  train 58 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.6/traces_brightclust_Nov2020_D0.6_set001.csv
  train 59 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.08/traces_brightclust_Nov2020_D0.08_set002.csv
  train 60 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.2/traces_brightclust_Nov2020_D0.2_set006.csv
  train 61 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.1/traces_brightclust_Nov2020_D0.1_set004.csv
  train 62 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.6/traces_brightclust_Nov2020_D0.6_set003.csv
  train 63 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.08/traces_brightclust_Nov2020_D0.08_set001.csv
  train 64 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.2/traces_brightclust_Nov2020_D0.2_set002.csv
  train 65 /beegfs/ye53nis/saves/firstartifact_Nov2020/10/traces_brightclust_Nov2020_D10_set009.csv
  train 66 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.6/traces_brightclust_Nov2020_D0.6_set008.csv
  train 67 /beegfs/ye53nis/saves/firstartifact_Nov2020/3.0/traces_brightclust_Nov2020_D3.0_set001.csv
  train 68 /beegfs/ye53nis/saves/firstartifact_Nov2020/50/traces_brightclust_Nov2020_D50_set002.csv
  train 69 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.2/traces_brightclust_Nov2020_D0.2_set003.csv
  train 70 /beegfs/ye53nis/saves/firstartifact_Nov2020/50/traces_brightclust_Nov2020_D50_set005.csv
  train 71 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.08/traces_brightclust_Nov2020_D0.08_set005.csv
  train 72 /beegfs/ye53nis/saves/firstartifact_Nov2020/10/traces_brightclust_Nov2020_D10_set004.csv
  train 73 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.08/traces_brightclust_Nov2020_D0.08_set010.csv
  train 74 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.1/traces_brightclust_Nov2020_D0.1_set010.csv
  train 75 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.4/traces_brightclust_Nov2020_D0.4_set010.csv
  train 76 /beegfs/ye53nis/saves/firstartifact_Nov2020/subsample/traces_brightclust_Nov2020_D0.2_set002.csv
  train 77 /beegfs/ye53nis/saves/firstartifact_Nov2020/subsample/traces_brightclust_Nov2020_D1.0_set003.csv
  train 78 /beegfs/ye53nis/saves/firstartifact_Nov2020/1.0/traces_brightclust_Nov2020_D1.0_set010.csv
  train 79 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.1/traces_brightclust_Nov2020_D0.1_set001.csv
  train 80 /beegfs/ye53nis/saves/firstartifact_Nov2020/50/traces_brightclust_Nov2020_D50_set009.csv
  train 81 /beegfs/ye53nis/saves/firstartifact_Nov2020/10/traces_brightclust_Nov2020_D10_set003.csv
  train 82 /beegfs/ye53nis/saves/firstartifact_Nov2020/10/traces_brightclust_Nov2020_D10_set008.csv
  test 83 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.1/traces_brightclust_Nov2020_D0.1_set006.csv
  test 84 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.2/traces_brightclust_Nov2020_D0.2_set008.csv
  test 85 /beegfs/ye53nis/saves/firstartifact_Nov2020/3.0/traces_brightclust_Nov2020_D3.0_set002.csv
  test 86 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.4/traces_brightclust_Nov2020_D0.4_set007.csv
  test 87 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.2/traces_brightclust_Nov2020_D0.2_set010.csv
  test 88 /beegfs/ye53nis/saves/firstartifact_Nov2020/1.0/traces_brightclust_Nov2020_D1.0_set006.csv
  test 89 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.6/traces_brightclust_Nov2020_D0.6_set009.csv
  test 90 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.08/traces_brightclust_Nov2020_D0.08_set003.csv
  test 91 /beegfs/ye53nis/saves/firstartifact_Nov2020/50/traces_brightclust_Nov2020_D50_set003.csv
  test 92 /beegfs/ye53nis/saves/firstartifact_Nov2020/3.0/traces_brightclust_Nov2020_D3.0_set009.csv
  test 93 /beegfs/ye53nis/saves/firstartifact_Nov2020/10/traces_brightclust_Nov2020_D10_set001.csv
  test 94 /beegfs/ye53nis/saves/firstartifact_Nov2020/3.0/traces_brightclust_Nov2020_D3.0_set008.csv
  test 95 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.2/traces_brightclust_Nov2020_D0.2_set007.csv
  test 96 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.1/traces_brightclust_Nov2020_D0.1_set002.csv
  test 97 /beegfs/ye53nis/saves/firstartifact_Nov2020/3.0/traces_brightclust_Nov2020_D3.0_set004.csv
  test 98 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.069/traces_brightclust_Nov2020_D0.069_set010.csv
  test 99 /beegfs/ye53nis/saves/firstartifact_Nov2020/subsample/traces_brightclust_Nov2020_D0.069_set001.csv
  test 100 /beegfs/ye53nis/saves/firstartifact_Nov2020/1.0/traces_brightclust_Nov2020_D1.0_set008.csv
  test 101 /beegfs/ye53nis/saves/firstartifact_Nov2020/1.0/traces_brightclust_Nov2020_D1.0_set005.csv
  test 102 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.4/traces_brightclust_Nov2020_D0.4_set008.csv
  test 103 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.4/traces_brightclust_Nov2020_D0.4_set005.csv
  
  The given DataFrame was split into 3 parts with shapes: [(16384, 8300), (16384, 8300), (16384, 8300)]
  The given DataFrame was split into 3 parts with shapes: [(16384, 2100), (16384, 2100), (16384, 2100)]
  
  for each 16384 timestap trace there are the following numbers of corrupted timesteps:
  label001_1    5916
  label002_1    7367
  label003_1     954
  label004_1    2965
  label005_1       0
  dtype: int64
  
  2021-02-05 18:31:32.060731: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
  To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
  
  number of training examples: 6640, number of validation examples: 1660
  
  ------------------------
  number of test examples: 2100
  
  input - shape:   (None, 16384, 1)
  output - shape:  (None, 16384, 1)
  
  2021-02-05 18:31:38.503252: I tensorflow/core/profiler/lib/profiler_session.cc:126] Profiler session initializing.
  2021-02-05 18:31:38.503297: I tensorflow/core/profiler/lib/profiler_session.cc:141] Profiler session started.
  2021-02-05 18:31:38.503361: I tensorflow/core/profiler/lib/profiler_session.cc:158] Profiler session tear down.
  2021/02/05 18:31:38 INFO mlflow.utils.autologging_utils: tensorflow autologging will track hyperparameters, performance metrics, model artifacts, and lineage information for the current tensorflow workflow to the MLflow run with ID 'b9935d1e554c423fb2852242f4c4504c'
  2021/02/05 18:31:38 WARNING mlflow.utils.autologging_utils: MLflow issued a warning during tensorflow autologging: "/home/ye53nis/.conda/envs/mlflow-fd9a200e5a24d4c79a0ff13be73ccb5141ed072c/lib/python3.8/site-packages/mlflow/utils/autologging_utils.py:86:
  UserWarning: Logging to MLflow failed: Changing param values is not allowed. Param with key='batch_size' was already logged with value='5' for run ID='b9935d1e554c423fb2852242f4c4504c'. Attempted logging new value 'None'."
  
  Epoch 1/50
  
  2021-02-05 18:31:49.433599: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:145] None of the MLIR Optimization Passes are enabled (registered 2)
  2021-02-05 18:31:49.686354: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 2300000000 Hz
  2021-02-05 18:31:57.757621: I tensorflow/core/profiler/lib/profiler_session.cc:126] Profiler session initializing.
  2021-02-05 18:31:57.757684: I tensorflow/core/profiler/lib/profiler_session.cc:141] Profiler session started.
  2021-02-05 18:31:59.478997: I tensorflow/core/profiler/lib/profiler_session.cc:66] Profiler session collecting data.
  2021-02-05 18:31:59.513272: I tensorflow/core/profiler/lib/profiler_session.cc:158] Profiler session tear down.
  2021-02-05 18:31:59.547737: I tensorflow/core/profiler/rpc/client/save_profile.cc:137] Creating directory: /tmp/tb/train/plugins/profile/2021_02_05_18_31_59
  2021-02-05 18:31:59.563056: I tensorflow/core/profiler/rpc/client/save_profile.cc:143] Dumped gzipped tool data for trace.json.gz to /tmp/tb/train/plugins/profile/2021_02_05_18_31_59/node221.trace.json.gz
  2021-02-05 18:31:59.600514: I tensorflow/core/profiler/rpc/client/save_profile.cc:137] Creating directory: /tmp/tb/train/plugins/profile/2021_02_05_18_31_59
  2021-02-05 18:31:59.600709: I tensorflow/core/profiler/rpc/client/save_profile.cc:143] Dumped gzipped tool data for memory_profile.json.gz to /tmp/tb/train/plugins/profile/2021_02_05_18_31_59/node221.memory_profile.json.gz
  2021-02-05 18:31:59.604669: I tensorflow/core/profiler/rpc/client/capture_profile.cc:251] Creating directory: /tmp/tb/train/plugins/profile/2021_02_05_18_31_59Dumped tool data for xplane.pb to /tmp/tb/train/plugins/profile/2021_02_05_18_31_59/node221.xplane.pb
  Dumped tool data for overview_page.pb to /tmp/tb/train/plugins/profile/2021_02_05_18_31_59/node221.overview_page.pb
  Dumped tool data for input_pipeline.pb to /tmp/tb/train/plugins/profile/2021_02_05_18_31_59/node221.input_pipeline.pb
  Dumped tool data for tensorflow_stats.pb to /tmp/tb/train/plugins/profile/2021_02_05_18_31_59/node221.tensorflow_stats.pb
  Dumped tool data for kernel_stats.pb to /tmp/tb/train/plugins/profile/2021_02_05_18_31_59/node221.kernel_stats.pb
  
  1280/1280 [==============================] - 2294s 2s/step - loss: 0.9535 - tp0.1: 7533469.6347 - fp0.1: 15687196.0695 - tn0.1: 28384947.3810 - fn0.1: 905043.5777 - precision0.1: 0.3192 - recall0.1: 0.8847 - tp0.3: 5299550.3950 - fp0.3
  : 6318015.6128 - tn0.3: 37754137.8610 - fn0.3: 3138962.8173 - precision0.3: 0.4501 - recall0.3: 0.6250 - tp0.5: 3326666.3817 - fp0.5: 1645942.8673 - tn0.5: 42426193.8962 - fn0.5: 5111846.8306 - precision0.5: 0.6413 - recall0.5: 0.3856
  - tp0.7: 2414052.7869 - fp0.7: 649878.6245 - tn0.7: 43422251.9945 - fn0.7: 6024460.4254 - precision0.7: 0.7630 - recall0.7: 0.2684 - tp0.9: 1333571.2881 - fp0.9: 122667.3622 - tn0.9: 43949483.5308 - fn0.9: 7104941.9243 - precision0.9:
  0.8933 - recall0.9: 0.1421 - accuracy: 0.8665 - auc: 0.8370 - val_loss: 4.5065 - val_tp0.1: 3953260.0000 - val_fp0.1: 22248736.0000 - val_tn0.1: 11977.0000 - val_fn0.1: 433.0000 - val_precision0.1: 0.1509 - val_recall0.1: 0.9999 - val_
  tp0.3: 3952790.0000 - val_fp0.3: 22240896.0000 - val_tn0.3: 19810.0000 - val_fn0.3: 903.0000 - val_precision0.3: 0.1509 - val_recall0.3: 0.9998 - val_tp0.5: 3932309.0000 - val_fp0.5: 18000460.0000 - val_tn0.5: 4260247.0000 - val_fn0.5:
   21384.0000 - val_precision0.5: 0.1793 - val_recall0.5: 0.9946 - val_tp0.7: 3907763.0000 - val_fp0.7: 15479875.0000 - val_tn0.7: 6780832.0000 - val_fn0.7: 45930.0000 - val_precision0.7: 0.2016 - val_recall0.7: 0.9884 - val_tp0.9: 38181
  41.0000 - val_fp0.9: 10776068.0000 - val_tn0.9: 11484639.0000 - val_fn0.9: 135552.0000 - val_precision0.9: 0.2616 - val_recall0.9: 0.9657 - val_accuracy: 0.3125 - val_auc: 0.8388
  Epoch 2/50
  1280/1280 [==============================] - 2266s 2s/step - loss: 0.7086 - tp0.1: 7408524.6518 - fp0.1: 9777539.1296 - tn0.1: 34354079.2826 - fn0.1: 970517.5371 - precision0.1: 0.4242 - recall0.1: 0.8837 - tp0.3: 6604603.1468 - fp0.3:
   5177327.2397 - tn0.3: 38954289.9594 - fn0.3: 1774439.0422 - precision0.3: 0.5497 - recall0.3: 0.7873 - tp0.5: 4795568.8735 - fp0.5: 1588506.0874 - tn0.5: 42543101.8407 - fn0.5: 3583473.3154 - precision0.5: 0.7554 - recall0.5: 0.5523 -
   tp0.7: 3842322.5410 - fp0.7: 585624.8493 - tn0.7: 43545999.7440 - fn0.7: 4536719.6479 - precision0.7: 0.8722 - recall0.7: 0.4455 - tp0.9: 2664836.5355 - fp0.9: 183975.7455 - tn0.9: 43947648.4012 - fn0.9: 5714205.6534 - precision0.9: 0
  .9372 - recall0.9: 0.3066 - accuracy: 0.9005 - auc: 0.8983 - val_loss: 8.8910 - val_tp0.1: 4309704.0000 - val_fp0.1: 21433558.0000 - val_tn0.1: 466616.0000 - val_fn0.1: 4509.0000 - val_precision0.1: 0.1674 - val_recall0.1: 0.9990 - val
  _tp0.3: 4304770.0000 - val_fp0.3: 20572086.0000 - val_tn0.3: 1328101.0000 - val_fn0.3: 9443.0000 - val_precision0.3: 0.1730 - val_recall0.3: 0.9978 - val_tp0.5: 4291421.0000 - val_fp0.5: 18429908.0000 - val_tn0.5: 3470280.0000 - val_fn
  0.5: 22792.0000 - val_precision0.5: 0.1889 - val_recall0.5: 0.9947 - val_tp0.7: 4286245.0000 - val_fp0.7: 17960968.0000 - val_tn0.7: 3939218.0000 - val_fn0.7: 27968.0000 - val_precision0.7: 0.1927 - val_recall0.7: 0.9935 - val_tp0.9: 4
  275720.0000 - val_fp0.9: 17168184.0000 - val_tn0.9: 4732003.0000 - val_fn0.9: 38493.0000 - val_precision0.9: 0.1994 - val_recall0.9: 0.9911 - val_accuracy: 0.2961 - val_auc: 0.6353
  Epoch 3/50
  1280/1280 [==============================] - 2269s 2s/step - loss: 0.4737 - tp0.1: 8043601.5324 - fp0.1: 6567924.1288 - tn0.1: 37316662.9188 - fn0.1: 582465.5137 - precision0.1: 0.5472 - recall0.1: 0.9292 - tp0.3: 7343577.1382 - fp0.3:
   2667019.8610 - tn0.3: 41217565.9859 - fn0.3: 1282489.9079 - precision0.3: 0.7338 - recall0.3: 0.8450 - tp0.5: 6431038.6987 - fp0.5: 1096394.8681 - tn0.5: 42788211.8212 - fn0.5: 2195028.3474 - precision0.5: 0.8653 - recall0.5: 0.7272 -
   tp0.7: 5761648.7400 - fp0.7: 554569.4169 - tn0.7: 43330028.4895 - fn0.7: 2864418.3060 - precision0.7: 0.9155 - recall0.7: 0.6553 - tp0.9: 4604418.9586 - fp0.9: 201930.2724 - tn0.9: 43682665.5402 - fn0.9: 4021648.0874 - precision0.9: 0
  .9590 - recall0.9: 0.5232 - accuracy: 0.9363 - auc: 0.9445 - val_loss: 0.6077 - val_tp0.1: 3651028.0000 - val_fp0.1: 3375728.0000 - val_tn0.1: 18695272.0000 - val_fn0.1: 492376.0000 - val_precision0.1: 0.5196 - val_recall0.1: 0.8812 -
  val_tp0.3: 3024227.0000 - val_fp0.3: 1523943.0000 - val_tn0.3: 20547044.0000 - val_fn0.3: 1119177.0000 - val_precision0.3: 0.6649 - val_recall0.3: 0.7299 - val_tp0.5: 2511712.0000 - val_fp0.5: 734684.0000 - val_tn0.5: 21336306.0000 - v
  al_fn0.5: 1631692.0000 - val_precision0.5: 0.7737 - val_recall0.5: 0.6062 - val_tp0.7: 2014285.0000 - val_fp0.7: 342510.0000 - val_tn0.7: 21728484.0000 - val_fn0.7: 2129119.0000 - val_precision0.7: 0.8547 - val_recall0.7: 0.4861 - val_
  tp0.9: 1375650.0000 - val_fp0.9: 83494.0000 - val_tn0.9: 21987508.0000 - val_fn0.9: 2767754.0000 - val_precision0.9: 0.9428 - val_recall0.9: 0.3320 - val_accuracy: 0.9097 - val_auc: 0.9057
  Epoch 4/50
  1280/1280 [==============================] - 2269s 2s/step - loss: 0.4104 - tp0.1: 8058950.6292 - fp0.1: 5549150.7541 - tn0.1: 38382155.2373 - fn0.1: 520398.2319 - precision0.1: 0.5890 - recall0.1: 0.9388 - tp0.3: 7465362.4442 - fp0.3:
   2368717.0968 - tn0.3: 41562596.7416 - fn0.3: 1113986.4169 - precision0.3: 0.7555 - recall0.3: 0.8690 - tp0.5: 6948225.4496 - fp0.5: 1288524.5277 - tn0.5: 42642762.4083 - fn0.5: 1631123.4114 - precision0.5: 0.8408 - recall0.5: 0.8075 -
   tp0.7: 6012711.4176 - fp0.7: 481306.9664 - tn0.7: 43449995.4871 - fn0.7: 2566637.4434 - precision0.7: 0.9250 - recall0.7: 0.6949 - tp0.9: 4953486.9508 - fp0.9: 170919.6136 - tn0.9: 43760406.4403 - fn0.9: 3625861.9102 - precision0.9: 0
  .9659 - recall0.9: 0.5674 - accuracy: 0.9430 - auc: 0.9540 - val_loss: 0.7360 - val_tp0.1: 3518565.0000 - val_fp0.1: 2418501.0000 - val_tn0.1: 19514842.0000 - val_fn0.1: 762498.0000 - val_precision0.1: 0.5926 - val_recall0.1: 0.8219 -
  val_tp0.3: 2934643.0000 - val_fp0.3: 1727197.0000 - val_tn0.3: 20206140.0000 - val_fn0.3: 1346420.0000 - val_precision0.3: 0.6295 - val_recall0.3: 0.6855 - val_tp0.5: 2401516.0000 - val_fp0.5: 1120115.0000 - val_tn0.5: 20813218.0000 -
  val_fn0.5: 1879547.0000 - val_precision0.5: 0.6819 - val_recall0.5: 0.5610 - val_tp0.7: 1995881.0000 - val_fp0.7: 587480.0000 - val_tn0.7: 21345854.0000 - val_fn0.7: 2285182.0000 - val_precision0.7: 0.7726 - val_recall0.7: 0.4662 - val
  _tp0.9: 1458232.0000 - val_fp0.9: 213624.0000 - val_tn0.9: 21719718.0000 - val_fn0.9: 2822831.0000 - val_precision0.9: 0.8722 - val_recall0.9: 0.3406 - val_accuracy: 0.8856 - val_auc: 0.8746
  Epoch 5/50
  1280/1280 [==============================] - 2256s 2s/step - loss: 0.3330 - tp0.1: 7934527.3903 - fp0.1: 4492667.3357 - tn0.1: 39650554.1023 - fn0.1: 432898.7760 - precision0.1: 0.6303 - recall0.1: 0.9481 - tp0.3: 7499415.2927 - fp0.3:
   1967988.6995 - tn0.3: 42175278.8080 - fn0.3: 868010.8735 - precision0.3: 0.7854 - recall0.3: 0.8951 - tp0.5: 7067133.4980 - fp0.5: 1049951.0960 - tn0.5: 43093262.6339 - fn0.5: 1300292.6682 - precision0.5: 0.8642 - recall0.5: 0.8435 -
  tp0.7: 6367827.1046 - fp0.7: 438987.0765 - tn0.7: 43704237.3872 - fn0.7: 1999599.0617 - precision0.7: 0.9322 - recall0.7: 0.7564 - tp0.9: 5357606.0351 - fp0.9: 138722.6315 - tn0.9: 44004504.8205 - fn0.9: 3009820.1311 - precision0.9: 0.
  9715 - recall0.9: 0.6358 - accuracy: 0.9541 - auc: 0.9638 - val_loss: 0.7327 - val_tp0.1: 3649063.0000 - val_fp0.1: 3036718.0000 - val_tn0.1: 18938916.0000 - val_fn0.1: 589701.0000 - val_precision0.1: 0.5458 - val_recall0.1: 0.8609 - v
  al_tp0.3: 3283999.0000 - val_fp0.3: 2440985.0000 - val_tn0.3: 19534644.0000 - val_fn0.3: 954765.0000 - val_precision0.3: 0.5736 - val_recall0.3: 0.7748 - val_tp0.5: 2951063.0000 - val_fp0.5: 2094966.0000 - val_tn0.5: 19880668.0000 - va
  l_fn0.5: 1287701.0000 - val_precision0.5: 0.5848 - val_recall0.5: 0.6962 - val_tp0.7: 2702547.0000 - val_fp0.7: 1708012.0000 - val_tn0.7: 20267626.0000 - val_fn0.7: 1536217.0000 - val_precision0.7: 0.6127 - val_recall0.7: 0.6376 - val_
  tp0.9: 2269870.0000 - val_fp0.9: 882080.0000 - val_tn0.9: 21093558.0000 - val_fn0.9: 1968894.0000 - val_precision0.9: 0.7201 - val_recall0.9: 0.5355 - val_accuracy: 0.8710 - val_auc: 0.8874
  Epoch 6/50
  1280/1280 [==============================] - 2267s 2s/step - loss: 0.3148 - tp0.1: 8107422.3794 - fp0.1: 4149459.1678 - tn0.1: 39840764.2201 - fn0.1: 412995.5535 - precision0.1: 0.6568 - recall0.1: 0.9505 - tp0.3: 7702537.1499 - fp0.3:
   1842434.6089 - tn0.3: 42147804.7213 - fn0.3: 817880.7830 - precision0.3: 0.8055 - recall0.3: 0.9021 - tp0.5: 7257172.3966 - fp0.5: 947340.6534 - tn0.5: 43042899.9469 - fn0.5: 1263245.5363 - precision0.5: 0.8838 - recall0.5: 0.8495 - t
  p0.7: 6767789.5652 - fp0.7: 485240.0351 - tn0.7: 43504992.3216 - fn0.7: 1752628.3677 - precision0.7: 0.9324 - recall0.7: 0.7924 - tp0.9: 5789730.9688 - fp0.9: 133595.3934 - tn0.9: 43856655.0180 - fn0.9: 2730686.9641 - precision0.9: 0.9
  771 - recall0.9: 0.6757 - accuracy: 0.9572 - auc: 0.9671 - val_loss: 0.5438 - val_tp0.1: 3469770.0000 - val_fp0.1: 1783021.0000 - val_tn0.1: 20362438.0000 - val_fn0.1: 599169.0000 - val_precision0.1: 0.6606 - val_recall0.1: 0.8527 - va
  l_tp0.3: 2981687.0000 - val_fp0.3: 776413.0000 - val_tn0.3: 21369044.0000 - val_fn0.3: 1087252.0000 - val_precision0.3: 0.7934 - val_recall0.3: 0.7328 - val_tp0.5: 2491815.0000 - val_fp0.5: 326150.0000 - val_tn0.5: 21819316.0000 - val_
  fn0.5: 1577124.0000 - val_precision0.5: 0.8843 - val_recall0.5: 0.6124 - val_tp0.7: 2092865.0000 - val_fp0.7: 128692.0000 - val_tn0.7: 22016760.0000 - val_fn0.7: 1976074.0000 - val_precision0.7: 0.9421 - val_recall0.7: 0.5144 - val_tp0
  .9: 1584013.0000 - val_fp0.9: 18793.0000 - val_tn0.9: 22126662.0000 - val_fn0.9: 2484926.0000 - val_precision0.9: 0.9883 - val_recall0.9: 0.3893 - val_accuracy: 0.9274 - val_auc: 0.9112
  Epoch 7/50
  1280/1280 [==============================] - 2250s 2s/step - loss: 0.2752 - tp0.1: 8215921.6979 - fp0.1: 3662051.7354 - tn0.1: 40263427.6151 - fn0.1: 369255.7354 - precision0.1: 0.6930 - recall0.1: 0.9578 - tp0.3: 7873789.5113 - fp0.3:
   1775915.5043 - tn0.3: 42149585.2553 - fn0.3: 711387.9219 - precision0.3: 0.8180 - recall0.3: 0.9178 - tp0.5: 7421898.0468 - fp0.5: 864234.9992 - tn0.5: 43061238.2006 - fn0.5: 1163279.3864 - precision0.5: 0.8970 - recall0.5: 0.8653 - t
  p0.7: 6899341.9555 - fp0.7: 403014.4684 - tn0.7: 43522441.7424 - fn0.7: 1685835.4778 - precision0.7: 0.9455 - recall0.7: 0.8048 - tp0.9: 6047520.0070 - fp0.9: 123672.6019 - tn0.9: 43801830.9766 - fn0.9: 2537657.4262 - precision0.9: 0.9
  803 - recall0.9: 0.7036 - accuracy: 0.9616 - auc: 0.9725 - val_loss: 0.5870 - val_tp0.1: 3992712.0000 - val_fp0.1: 3100961.0000 - val_tn0.1: 18785236.0000 - val_fn0.1: 335494.0000 - val_precision0.1: 0.5629 - val_recall0.1: 0.9225 - va
  l_tp0.3: 3697613.0000 - val_fp0.3: 2362999.0000 - val_tn0.3: 19523192.0000 - val_fn0.3: 630593.0000 - val_precision0.3: 0.6101 - val_recall0.3: 0.8543 - val_tp0.5: 3356073.0000 - val_fp0.5: 1926159.0000 - val_tn0.5: 19960034.0000 - val
  _fn0.5: 972133.0000 - val_precision0.5: 0.6354 - val_recall0.5: 0.7754 - val_tp0.7: 3025256.0000 - val_fp0.7: 1584253.0000 - val_tn0.7: 20301946.0000 - val_fn0.7: 1302950.0000 - val_precision0.7: 0.6563 - val_recall0.7: 0.6990 - val_tp
  0.9: 2583002.0000 - val_fp0.9: 1016388.0000 - val_tn0.9: 20869810.0000 - val_fn0.9: 1745204.0000 - val_precision0.9: 0.7176 - val_recall0.9: 0.5968 - val_accuracy: 0.8894 - val_auc: 0.9219
  Epoch 8/50
  1280/1280 [==============================] - 2264s 2s/step - loss: 0.2659 - tp0.1: 8145147.1616 - fp0.1: 3529957.8525 - tn0.1: 40486117.0164 - fn0.1: 349436.3614 - precision0.1: 0.6977 - recall0.1: 0.9588 - tp0.3: 7816330.9524 - fp0.3:
   1688486.8493 - tn0.3: 42327603.8790 - fn0.3: 678252.5706 - precision0.3: 0.8221 - recall0.3: 0.9201 - tp0.5: 7396513.9524 - fp0.5: 842774.1983 - tn0.5: 43173300.2077 - fn0.5: 1098069.5706 - precision0.5: 0.8972 - recall0.5: 0.8704 - t
  p0.7: 6908332.2162 - fp0.7: 399714.5441 - tn0.7: 43616338.9617 - fn0.7: 1586251.3068 - precision0.7: 0.9448 - recall0.7: 0.8128 - tp0.9: 6106920.7744 - fp0.9: 123699.0000 - tn0.9: 43892389.6027 - fn0.9: 2387662.7486 - precision0.9: 0.9
  796 - recall0.9: 0.7174 - accuracy: 0.9627 - auc: 0.9735 - val_loss: 0.5059 - val_tp0.1: 3832991.0000 - val_fp0.1: 2975127.0000 - val_tn0.1: 19105348.0000 - val_fn0.1: 300933.0000 - val_precision0.1: 0.5630 - val_recall0.1: 0.9272 - va
  l_tp0.3: 3565080.0000 - val_fp0.3: 2148056.0000 - val_tn0.3: 19932426.0000 - val_fn0.3: 568844.0000 - val_precision0.3: 0.6240 - val_recall0.3: 0.8624 - val_tp0.5: 3244053.0000 - val_fp0.5: 1637048.0000 - val_tn0.5: 20443428.0000 - val
  _fn0.5: 889871.0000 - val_precision0.5: 0.6646 - val_recall0.5: 0.7847 - val_tp0.7: 2923895.0000 - val_fp0.7: 1240875.0000 - val_tn0.7: 20839598.0000 - val_fn0.7: 1210029.0000 - val_precision0.7: 0.7021 - val_recall0.7: 0.7073 - val_tp
  0.9: 2501710.0000 - val_fp0.9: 630185.0000 - val_tn0.9: 21450288.0000 - val_fn0.9: 1632214.0000 - val_precision0.9: 0.7988 - val_recall0.9: 0.6052 - val_accuracy: 0.9036 - val_auc: 0.9333
  Epoch 9/50
  1280/1280 [==============================] - 2265s 2s/step - loss: 0.2721 - tp0.1: 8177633.5020 - fp0.1: 3535203.1725 - tn0.1: 40418482.1335 - fn0.1: 379327.7697 - precision0.1: 0.7002 - recall0.1: 0.9554 - tp0.3: 7847088.7190 - fp0.3:
   1695397.4301 - tn0.3: 42258294.9282 - fn0.3: 709872.5527 - precision0.3: 0.8241 - recall0.3: 0.9169 - tp0.5: 7427192.4395 - fp0.5: 834997.2678 - tn0.5: 43118702.8587 - fn0.5: 1129768.8322 - precision0.5: 0.9006 - recall0.5: 0.8679 - t
  p0.7: 6938678.2849 - fp0.7: 395857.1007 - tn0.7: 43557853.7744 - fn0.7: 1618282.9867 - precision0.7: 0.9471 - recall0.7: 0.8106 - tp0.9: 6133368.2951 - fp0.9: 120955.7010 - tn0.9: 43832745.2506 - fn0.9: 2423592.9766 - precision0.9: 0.9
  811 - recall0.9: 0.7170 - accuracy: 0.9631 - auc: 0.9718 - val_loss: 0.5832 - val_tp0.1: 3893714.0000 - val_fp0.1: 3303616.0000 - val_tn0.1: 18681092.0000 - val_fn0.1: 335977.0000 - val_precision0.1: 0.5410 - val_recall0.1: 0.9206 - va
  l_tp0.3: 3627551.0000 - val_fp0.3: 2498540.0000 - val_tn0.3: 19486168.0000 - val_fn0.3: 602140.0000 - val_precision0.3: 0.5921 - val_recall0.3: 0.8576 - val_tp0.5: 3322248.0000 - val_fp0.5: 1987125.0000 - val_tn0.5: 19997588.0000 - val
  _fn0.5: 907443.0000 - val_precision0.5: 0.6257 - val_recall0.5: 0.7855 - val_tp0.7: 3013599.0000 - val_fp0.7: 1581768.0000 - val_tn0.7: 20402948.0000 - val_fn0.7: 1216092.0000 - val_precision0.7: 0.6558 - val_recall0.7: 0.7125 - val_tp
  0.9: 2613832.0000 - val_fp0.9: 924572.0000 - val_tn0.9: 21060124.0000 - val_fn0.9: 1615859.0000 - val_precision0.9: 0.7387 - val_recall0.9: 0.6180 - val_accuracy: 0.8896 - val_auc: 0.9233
  Epoch 10/50
  1280/1280 [==============================] - 2167s 2s/step - loss: 0.2587 - tp0.1: 8173202.1491 - fp0.1: 3419344.5308 - tn0.1: 40568773.3271 - fn0.1: 349346.8673 - precision0.1: 0.7050 - recall0.1: 0.9592 - tp0.3: 7845280.8431 - fp0.3:
   1633368.1054 - tn0.3: 42354746.7455 - fn0.3: 677268.1733 - precision0.3: 0.8274 - recall0.3: 0.9210 - tp0.5: 7444574.4614 - fp0.5: 808848.8946 - tn0.5: 43179274.2358 - fn0.5: 1077974.5550 - precision0.5: 0.9017 - recall0.5: 0.8744 - t
  p0.7: 6974399.5706 - fp0.7: 382341.6963 - tn0.7: 43605795.4044 - fn0.7: 1548149.4457 - precision0.7: 0.9478 - recall0.7: 0.8197 - tp0.9: 6212727.7385 - fp0.9: 117973.7447 - tn0.9: 43870128.1835 - fn0.9: 2309821.2779 - precision0.9: 0.9
  813 - recall0.9: 0.7300 - accuracy: 0.9641 - auc: 0.9742 - val_loss: 0.6408 - val_tp0.1: 3921454.0000 - val_fp0.1: 3165544.0000 - val_tn0.1: 18831180.0000 - val_fn0.1: 296224.0000 - val_precision0.1: 0.5533 - val_recall0.1: 0.9298 - va
  l_tp0.3: 3674710.0000 - val_fp0.3: 2541189.0000 - val_tn0.3: 19455532.0000 - val_fn0.3: 542968.0000 - val_precision0.3: 0.5912 - val_recall0.3: 0.8713 - val_tp0.5: 3370841.0000 - val_fp0.5: 2184413.0000 - val_tn0.5: 19812308.0000 - val
  _fn0.5: 846837.0000 - val_precision0.5: 0.6068 - val_recall0.5: 0.7992 - val_tp0.7: 3057156.0000 - val_fp0.7: 1929320.0000 - val_tn0.7: 20067408.0000 - val_fn0.7: 1160522.0000 - val_precision0.7: 0.6131 - val_recall0.7: 0.7248 - val_tp
  0.9: 2691235.0000 - val_fp0.9: 1545100.0000 - val_tn0.9: 20451626.0000 - val_fn0.9: 1526443.0000 - val_precision0.9: 0.6353 - val_recall0.9: 0.6381 - val_accuracy: 0.8844 - val_auc: 0.9185
  Epoch 11/50
  1280/1280 [==============================] - 2195s 2s/step - loss: 0.2597 - tp0.1: 8080787.5496 - fp0.1: 3464173.5402 - tn0.1: 40607994.5691 - fn0.1: 357720.7119 - precision0.1: 0.6992 - recall0.1: 0.9583 - tp0.3: 7754354.7728 - fp0.3:
   1652487.9594 - tn0.3: 42419634.1717 - fn0.3: 684153.4887 - precision0.3: 0.8245 - recall0.3: 0.9197 - tp0.5: 7350582.6308 - fp0.5: 814018.4028 - tn0.5: 43258092.7291 - fn0.5: 1087925.6308 - precision0.5: 0.9008 - recall0.5: 0.8720 - t
  p0.7: 6874278.2834 - fp0.7: 383492.2842 - tn0.7: 43688666.7549 - fn0.7: 1564229.9781 - precision0.7: 0.9477 - recall0.7: 0.8158 - tp0.9: 6117529.6448 - fp0.9: 121059.4980 - tn0.9: 43951099.9110 - fn0.9: 2320978.6167 - precision0.9: 0.9
  810 - recall0.9: 0.7266 - accuracy: 0.9642 - auc: 0.9736 - val_loss: 0.6007 - val_tp0.1: 3926588.0000 - val_fp0.1: 3104163.0000 - val_tn0.1: 18816186.0000 - val_fn0.1: 367465.0000 - val_precision0.1: 0.5585 - val_recall0.1: 0.9144 - va
  l_tp0.3: 3638712.0000 - val_fp0.3: 2431931.0000 - val_tn0.3: 19488412.0000 - val_fn0.3: 655341.0000 - val_precision0.3: 0.5994 - val_recall0.3: 0.8474 - val_tp0.5: 3274020.0000 - val_fp0.5: 1937942.0000 - val_tn0.5: 19982404.0000 - val
  _fn0.5: 1020033.0000 - val_precision0.5: 0.6282 - val_recall0.5: 0.7625 - val_tp0.7: 2918102.0000 - val_fp0.7: 1553208.0000 - val_tn0.7: 20367136.0000 - val_fn0.7: 1375951.0000 - val_precision0.7: 0.6526 - val_recall0.7: 0.6796 - val_t
  p0.9: 2486914.0000 - val_fp0.9: 949847.0000 - val_tn0.9: 20970500.0000 - val_fn0.9: 1807139.0000 - val_precision0.9: 0.7236 - val_recall0.9: 0.5792 - val_accuracy: 0.8872 - val_auc: 0.9170
  Epoch 12/50
  1280/1280 [==============================] - 2210s 2s/step - loss: 0.2529 - tp0.1: 8177778.9118 - fp0.1: 3446344.3888 - tn0.1: 40554141.6956 - fn0.1: 332395.2459 - precision0.1: 0.7024 - recall0.1: 0.9615 - tp0.3: 7866160.7268 - fp0.3:
   1663533.3247 - tn0.3: 42336942.1569 - fn0.3: 644013.4309 - precision0.3: 0.8244 - recall0.3: 0.9251 - tp0.5: 7468651.7650 - fp0.5: 826902.3630 - tn0.5: 43173597.9063 - fn0.5: 1041522.3927 - precision0.5: 0.8996 - recall0.5: 0.8783 - t
  p0.7: 6982137.1124 - fp0.7: 384836.8439 - tn0.7: 43615637.7978 - fn0.7: 1528037.0453 - precision0.7: 0.9473 - recall0.7: 0.8206 - tp0.9: 6215436.7151 - fp0.9: 115782.2108 - tn0.9: 43884692.7596 - fn0.9: 2294737.4426 - precision0.9: 0.9
  816 - recall0.9: 0.7291 - accuracy: 0.9644 - auc: 0.9753 - val_loss: 0.5433 - val_tp0.1: 3996675.0000 - val_fp0.1: 3049375.0000 - val_tn0.1: 18853738.0000 - val_fn0.1: 314612.0000 - val_precision0.1: 0.5672 - val_recall0.1: 0.9270 - va
  l_tp0.3: 3730606.0000 - val_fp0.3: 2336826.0000 - val_tn0.3: 19566286.0000 - val_fn0.3: 580681.0000 - val_precision0.3: 0.6149 - val_recall0.3: 0.8653 - val_tp0.5: 3416555.0000 - val_fp0.5: 1856264.0000 - val_tn0.5: 20046848.0000 - val
  _fn0.5: 894732.0000 - val_precision0.5: 0.6480 - val_recall0.5: 0.7925 - val_tp0.7: 3097348.0000 - val_fp0.7: 1461441.0000 - val_tn0.7: 20441664.0000 - val_fn0.7: 1213939.0000 - val_precision0.7: 0.6794 - val_recall0.7: 0.7184 - val_tp
  0.9: 2698173.0000 - val_fp0.9: 854641.0000 - val_tn0.9: 21048466.0000 - val_fn0.9: 1613114.0000 - val_precision0.9: 0.7594 - val_recall0.9: 0.6258 - val_accuracy: 0.8951 - val_auc: 0.9303
  Epoch 13/50
  1280/1280 [==============================] - 2209s 2s/step - loss: 0.2538 - tp0.1: 8165544.5761 - fp0.1: 3360321.7119 - tn0.1: 40640034.6550 - fn0.1: 344752.3396 - precision0.1: 0.7106 - recall0.1: 0.9599 - tp0.3: 7853592.0312 - fp0.3:
   1646282.5488 - tn0.3: 42354071.1335 - fn0.3: 656704.8845 - precision0.3: 0.8279 - recall0.3: 0.9236 - tp0.5: 7458424.3138 - fp0.5: 826341.5082 - tn0.5: 43174017.1226 - fn0.5: 1051872.6019 - precision0.5: 0.9009 - recall0.5: 0.8775 - t
  p0.7: 6979820.3271 - fp0.7: 395956.8899 - tn0.7: 43604407.7775 - fn0.7: 1530476.5886 - precision0.7: 0.9468 - recall0.7: 0.8215 - tp0.9: 6232427.8197 - fp0.9: 129181.2912 - tn0.9: 43871195.4387 - fn0.9: 2277869.0960 - precision0.9: 0.9
  799 - recall0.9: 0.7346 - accuracy: 0.9644 - auc: 0.9747 - val_loss: 0.5485 - val_tp0.1: 3997124.0000 - val_fp0.1: 3403162.0000 - val_tn0.1: 18545812.0000 - val_fn0.1: 268302.0000 - val_precision0.1: 0.5401 - val_recall0.1: 0.9371 - va
  l_tp0.3: 3749177.0000 - val_fp0.3: 2616473.0000 - val_tn0.3: 19332510.0000 - val_fn0.3: 516249.0000 - val_precision0.3: 0.5890 - val_recall0.3: 0.8790 - val_tp0.5: 3440083.0000 - val_fp0.5: 2075288.0000 - val_tn0.5: 19873688.0000 - val
  _fn0.5: 825343.0000 - val_precision0.5: 0.6237 - val_recall0.5: 0.8065 - val_tp0.7: 3117260.0000 - val_fp0.7: 1623393.0000 - val_tn0.7: 20325572.0000 - val_fn0.7: 1148166.0000 - val_precision0.7: 0.6576 - val_recall0.7: 0.7308 - val_tp
  0.9: 2708258.0000 - val_fp0.9: 926738.0000 - val_tn0.9: 21022226.0000 - val_fn0.9: 1557168.0000 - val_precision0.9: 0.7451 - val_recall0.9: 0.6349 - val_accuracy: 0.8894 - val_auc: 0.9322
  Epoch 14/50
  1280/1280 [==============================] - 2223s 2s/step - loss: 0.2483 - tp0.1: 8105005.2256 - fp0.1: 3301412.7728 - tn0.1: 40770974.7994 - fn0.1: 333249.2350 - precision0.1: 0.7102 - recall0.1: 0.9606 - tp0.3: 7797681.4614 - fp0.3:
   1632480.2381 - tn0.3: 42439931.4738 - fn0.3: 640572.9992 - precision0.3: 0.8273 - recall0.3: 0.9241 - tp0.5: 7399965.9297 - fp0.5: 804918.4660 - tn0.5: 43267490.8134 - fn0.5: 1038288.5308 - precision0.5: 0.9020 - recall0.5: 0.8772 - t
  p0.7: 6921871.3716 - fp0.7: 370916.4372 - tn0.7: 43701494.4621 - fn0.7: 1516383.0890 - precision0.7: 0.9493 - recall0.7: 0.8208 - tp0.9: 6187238.5597 - fp0.9: 114174.0468 - tn0.9: 43958233.5792 - fn0.9: 2251015.9009 - precision0.9: 0.9
  819 - recall0.9: 0.7337 - accuracy: 0.9651 - auc: 0.9752 - val_loss: 0.4990 - val_tp0.1: 3920222.0000 - val_fp0.1: 2895972.0000 - val_tn0.1: 19106396.0000 - val_fn0.1: 291803.0000 - val_precision0.1: 0.5751 - val_recall0.1: 0.9307 - va
  l_tp0.3: 3647799.0000 - val_fp0.3: 2161341.0000 - val_tn0.3: 19841040.0000 - val_fn0.3: 564226.0000 - val_precision0.3: 0.6279 - val_recall0.3: 0.8660 - val_tp0.5: 3322418.0000 - val_fp0.5: 1638347.0000 - val_tn0.5: 20364020.0000 - val
  _fn0.5: 889607.0000 - val_precision0.5: 0.6697 - val_recall0.5: 0.7888 - val_tp0.7: 2985353.0000 - val_fp0.7: 1222889.0000 - val_tn0.7: 20779488.0000 - val_fn0.7: 1226672.0000 - val_precision0.7: 0.7094 - val_recall0.7: 0.7088 - val_tp
  0.9: 2563095.0000 - val_fp0.9: 657853.0000 - val_tn0.9: 21344516.0000 - val_fn0.9: 1648930.0000 - val_precision0.9: 0.7958 - val_recall0.9: 0.6085 - val_accuracy: 0.9036 - val_auc: 0.9356
  Epoch 15/50
  1280/1280 [==============================] - 2223s 2s/step - loss: 0.2505 - tp0.1: 8129718.4130 - fp0.1: 3285117.6674 - tn0.1: 40755981.7447 - fn0.1: 339847.1585 - precision0.1: 0.7126 - recall0.1: 0.9599 - tp0.3: 7829732.9274 - fp0.3:
   1632416.1491 - tn0.3: 42408661.6440 - fn0.3: 639832.6440 - precision0.3: 0.8279 - recall0.3: 0.9244 - tp0.5: 7437810.6112 - fp0.5: 808981.5230 - tn0.5: 43232126.9953 - fn0.5: 1031754.9602 - precision0.5: 0.9024 - recall0.5: 0.8777 - t
  p0.7: 6957910.6557 - fp0.7: 373871.2498 - tn0.7: 43667250.3630 - fn0.7: 1511654.9157 - precision0.7: 0.9493 - recall0.7: 0.8208 - tp0.9: 6212919.2295 - fp0.9: 115091.8977 - tn0.9: 43925968.1874 - fn0.9: 2256646.3419 - precision0.9: 0.9
  817 - recall0.9: 0.7317 - accuracy: 0.9649 - auc: 0.9750 - val_loss: 0.5413 - val_tp0.1: 3929730.0000 - val_fp0.1: 3234417.0000 - val_tn0.1: 18748682.0000 - val_fn0.1: 301573.0000 - val_precision0.1: 0.5485 - val_recall0.1: 0.9287 - va
  l_tp0.3: 3652364.0000 - val_fp0.3: 2547235.0000 - val_tn0.3: 19435862.0000 - val_fn0.3: 578939.0000 - val_precision0.3: 0.5891 - val_recall0.3: 0.8632 - val_tp0.5: 3294284.0000 - val_fp0.5: 1952001.0000 - val_tn0.5: 20031108.0000 - val
  _fn0.5: 937019.0000 - val_precision0.5: 0.6279 - val_recall0.5: 0.7786 - val_tp0.7: 2934269.0000 - val_fp0.7: 1461182.0000 - val_tn0.7: 20521920.0000 - val_fn0.7: 1297034.0000 - val_precision0.7: 0.6676 - val_recall0.7: 0.6935 - val_tp
  0.9: 2494237.0000 - val_fp0.9: 748300.0000 - val_tn0.9: 21234804.0000 - val_fn0.9: 1737066.0000 - val_precision0.9: 0.7692 - val_recall0.9: 0.5895 - val_accuracy: 0.8898 - val_auc: 0.9282
  Epoch 16/50
  1280/1280 [==============================] - 2215s 2s/step - loss: 0.2478 - tp0.1: 8105796.2350 - fp0.1: 3240299.5347 - tn0.1: 40826166.4356 - fn0.1: 338386.8126 - precision0.1: 0.7132 - recall0.1: 0.9597 - tp0.3: 7813940.3685 - fp0.3:
   1622845.5285 - tn0.3: 42443650.8525 - fn0.3: 630242.6792 - precision0.3: 0.8274 - recall0.3: 0.9252 - tp0.5: 7422996.8228 - fp0.5: 793661.0414 - tn0.5: 43272824.2966 - fn0.5: 1021186.2248 - precision0.5: 0.9030 - recall0.5: 0.8785 - t
  p0.7: 6948059.6292 - fp0.7: 366674.7080 - tn0.7: 43699824.9578 - fn0.7: 1496123.4184 - precision0.7: 0.9495 - recall0.7: 0.8219 - tp0.9: 6225846.1991 - fp0.9: 114013.1436 - tn0.9: 43952461.9219 - fn0.9: 2218336.8486 - precision0.9: 0.9
  817 - recall0.9: 0.7370 - accuracy: 0.9654 - auc: 0.9750 - val_loss: 0.5514 - val_tp0.1: 3985944.0000 - val_fp0.1: 3204062.0000 - val_tn0.1: 18714132.0000 - val_fn0.1: 310258.0000 - val_precision0.1: 0.5544 - val_recall0.1: 0.9278 - va
  l_tp0.3: 3706293.0000 - val_fp0.3: 2449108.0000 - val_tn0.3: 19469088.0000 - val_fn0.3: 589909.0000 - val_precision0.3: 0.6021 - val_recall0.3: 0.8627 - val_tp0.5: 3369179.0000 - val_fp0.5: 1882360.0000 - val_tn0.5: 20035840.0000 - val
  _fn0.5: 927023.0000 - val_precision0.5: 0.6416 - val_recall0.5: 0.7842 - val_tp0.7: 3033922.0000 - val_fp0.7: 1434846.0000 - val_tn0.7: 20483356.0000 - val_fn0.7: 1262280.0000 - val_precision0.7: 0.6789 - val_recall0.7: 0.7062 - val_tp
  0.9: 2621404.0000 - val_fp0.9: 840233.0000 - val_tn0.9: 21077966.0000 - val_fn0.9: 1674798.0000 - val_precision0.9: 0.7573 - val_recall0.9: 0.6102 - val_accuracy: 0.8928 - val_auc: 0.9287
  Epoch 17/50
  1280/1280 [==============================] - 2214s 2s/step - loss: 0.2484 - tp0.1: 8227665.5144 - fp0.1: 3202591.7237 - tn0.1: 40740266.8517 - fn0.1: 340122.5121 - precision0.1: 0.7214 - recall0.1: 0.9603 - tp0.3: 7941320.1460 - fp0.3:
   1633513.9258 - tn0.3: 42309362.5909 - fn0.3: 626467.8806 - precision0.3: 0.8301 - recall0.3: 0.9272 - tp0.5: 7558350.9258 - fp0.5: 804203.6698 - tn0.5: 43138639.2201 - fn0.5: 1009437.1007 - precision0.5: 0.9041 - recall0.5: 0.8831 - t
  p0.7: 7083529.7642 - fp0.7: 372694.8454 - tn0.7: 43570150.8829 - fn0.7: 1484258.2623 - precision0.7: 0.9501 - recall0.7: 0.8284 - tp0.9: 6341241.1249 - fp0.9: 113794.4754 - tn0.9: 43829043.6105 - fn0.9: 2226546.9016 - precision0.9: 0.9
  824 - recall0.9: 0.7414 - accuracy: 0.9652 - auc: 0.9756 - val_loss: 0.5142 - val_tp0.1: 3628279.0000 - val_fp0.1: 2753615.0000 - val_tn0.1: 19485962.0000 - val_fn0.1: 346542.0000 - val_precision0.1: 0.5685 - val_recall0.1: 0.9128 - va
  l_tp0.3: 3324088.0000 - val_fp0.3: 2056425.0000 - val_tn0.3: 20183156.0000 - val_fn0.3: 650733.0000 - val_precision0.3: 0.6178 - val_recall0.3: 0.8363 - val_tp0.5: 2972065.0000 - val_fp0.5: 1467191.0000 - val_tn0.5: 20772396.0000 - val
  _fn0.5: 1002756.0000 - val_precision0.5: 0.6695 - val_recall0.5: 0.7477 - val_tp0.7: 2650974.0000 - val_fp0.7: 1083574.0000 - val_tn0.7: 21156004.0000 - val_fn0.7: 1323847.0000 - val_precision0.7: 0.7099 - val_recall0.7: 0.6669 - val_t
  p0.9: 2300149.0000 - val_fp0.9: 585684.0000 - val_tn0.9: 21653888.0000 - val_fn0.9: 1674672.0000 - val_precision0.9: 0.7970 - val_recall0.9: 0.5787 - val_accuracy: 0.9058 - val_auc: 0.9268
  Epoch 18/50
  1280/1280 [==============================] - 2223s 2s/step - loss: 0.2451 - tp0.1: 8257483.8283 - fp0.1: 3209709.3536 - tn0.1: 40707026.2334 - fn0.1: 336440.2756 - precision0.1: 0.7174 - recall0.1: 0.9618 - tp0.3: 7972148.5215 - fp0.3:
   1611985.2998 - tn0.3: 42304753.9649 - fn0.3: 621775.5824 - precision0.3: 0.8295 - recall0.3: 0.9289 - tp0.5: 7592991.5839 - fp0.5: 801375.8977 - tn0.5: 43115384.3333 - fn0.5: 1000932.5199 - precision0.5: 0.9027 - recall0.5: 0.8847 - t
  p0.7: 7121459.2779 - fp0.7: 371705.6339 - tn0.7: 43545020.0328 - fn0.7: 1472464.8259 - precision0.7: 0.9492 - recall0.7: 0.8297 - tp0.9: 6381060.6097 - fp0.9: 111751.2303 - tn0.9: 43804981.0523 - fn0.9: 2212863.4941 - precision0.9: 0.9
  823 - recall0.9: 0.7435 - accuracy: 0.9654 - auc: 0.9764 - val_loss: 0.5303 - val_tp0.1: 3950393.0000 - val_fp0.1: 2825498.0000 - val_tn0.1: 19094474.0000 - val_fn0.1: 344035.0000 - val_precision0.1: 0.5830 - val_recall0.1: 0.9199 - va
  l_tp0.3: 3645339.0000 - val_fp0.3: 2077885.0000 - val_tn0.3: 19842076.0000 - val_fn0.3: 649089.0000 - val_precision0.3: 0.6369 - val_recall0.3: 0.8489 - val_tp0.5: 3283330.0000 - val_fp0.5: 1558246.0000 - val_tn0.5: 20361722.0000 - val
  _fn0.5: 1011098.0000 - val_precision0.5: 0.6782 - val_recall0.5: 0.7646 - val_tp0.7: 2947342.0000 - val_fp0.7: 1176365.0000 - val_tn0.7: 20743604.0000 - val_fn0.7: 1347086.0000 - val_precision0.7: 0.7147 - val_recall0.7: 0.6863 - val_t
  p0.9: 2536959.0000 - val_fp0.9: 698850.0000 - val_tn0.9: 21221120.0000 - val_fn0.9: 1757469.0000 - val_precision0.9: 0.7840 - val_recall0.9: 0.5908 - val_accuracy: 0.9020 - val_auc: 0.9295
  Epoch 19/50
  1280/1280 [==============================] - 2211s 2s/step - loss: 0.2463 - tp0.1: 8262233.3575 - fp0.1: 3162400.5699 - tn0.1: 40752275.6565 - fn0.1: 333748.0101 - precision0.1: 0.7238 - recall0.1: 0.9607 - tp0.3: 7976781.4278 - fp0.3:
   1621499.8923 - tn0.3: 42293181.6620 - fn0.3: 619199.9399 - precision0.3: 0.8316 - recall0.3: 0.9269 - tp0.5: 7600963.2264 - fp0.5: 819732.1842 - tn0.5: 43094936.0523 - fn0.5: 995018.1413 - precision0.5: 0.9028 - recall0.5: 0.8828 - tp
  0.7: 7125748.6019 - fp0.7: 384384.1327 - tn0.7: 43530286.4083 - fn0.7: 1470232.7658 - precision0.7: 0.9489 - recall0.7: 0.8275 - tp0.9: 6355596.0687 - fp0.9: 116562.7260 - tn0.9: 43798125.0718 - fn0.9: 2240385.2990 - precision0.9: 0.98
  20 - recall0.9: 0.7377 - accuracy: 0.9650 - auc: 0.9759 - val_loss: 0.5854 - val_tp0.1: 3832029.0000 - val_fp0.1: 3030240.0000 - val_tn0.1: 18976838.0000 - val_fn0.1: 375292.0000 - val_precision0.1: 0.5584 - val_recall0.1: 0.9108 - val
  _tp0.3: 3499789.0000 - val_fp0.3: 2341510.0000 - val_tn0.3: 19665576.0000 - val_fn0.3: 707532.0000 - val_precision0.3: 0.5991 - val_recall0.3: 0.8318 - val_tp0.5: 3162666.0000 - val_fp0.5: 1856596.0000 - val_tn0.5: 20150478.0000 - val_
  fn0.5: 1044655.0000 - val_precision0.5: 0.6301 - val_recall0.5: 0.7517 - val_tp0.7: 2854187.0000 - val_fp0.7: 1428102.0000 - val_tn0.7: 20578972.0000 - val_fn0.7: 1353134.0000 - val_precision0.7: 0.6665 - val_recall0.7: 0.6784 - val_tp
  0.9: 2488186.0000 - val_fp0.9: 879408.0000 - val_tn0.9: 21127672.0000 - val_fn0.9: 1719135.0000 - val_precision0.9: 0.7389 - val_recall0.9: 0.5914 - val_accuracy: 0.8893 - val_auc: 0.9201
  Epoch 20/50
  1280/1280 [==============================] - 2220s 2s/step - loss: 0.2304 - tp0.1: 8074195.9617 - fp0.1: 2966270.8033 - tn0.1: 41146339.4450 - fn0.1: 323842.2264 - precision0.1: 0.7319 - recall0.1: 0.9616 - tp0.3: 7805955.5699 - fp0.3:
   1533470.4364 - tn0.3: 42579145.3279 - fn0.3: 592082.6183 - precision0.3: 0.8364 - recall0.3: 0.9297 - tp0.5: 7448954.0062 - fp0.5: 779671.7213 - tn0.5: 43332921.9258 - fn0.5: 949084.1819 - precision0.5: 0.9056 - recall0.5: 0.8873 - tp
  0.7: 6997774.1819 - fp0.7: 363044.1397 - tn0.7: 43749575.7775 - fn0.7: 1400264.0062 - precision0.7: 0.9510 - recall0.7: 0.8336 - tp0.9: 6267801.5144 - fp0.9: 107614.1304 - tn0.9: 44004989.1429 - fn0.9: 2130236.6737 - precision0.9: 0.98
  33 - recall0.9: 0.7471 - accuracy: 0.9673 - auc: 0.9771 - val_loss: 0.5968 - val_tp0.1: 3918274.0000 - val_fp0.1: 3250251.0000 - val_tn0.1: 18736068.0000 - val_fn0.1: 309803.0000 - val_precision0.1: 0.5466 - val_recall0.1: 0.9267 - val
  _tp0.3: 3617081.0000 - val_fp0.3: 2559029.0000 - val_tn0.3: 19427288.0000 - val_fn0.3: 610996.0000 - val_precision0.3: 0.5857 - val_recall0.3: 0.8555 - val_tp0.5: 3277656.0000 - val_fp0.5: 2084096.0000 - val_tn0.5: 19902236.0000 - val_
  fn0.5: 950421.0000 - val_precision0.5: 0.6113 - val_recall0.5: 0.7752 - val_tp0.7: 2954727.0000 - val_fp0.7: 1696039.0000 - val_tn0.7: 20290280.0000 - val_fn0.7: 1273350.0000 - val_precision0.7: 0.6353 - val_recall0.7: 0.6988 - val_tp0
  .9: 2580804.0000 - val_fp0.9: 1147305.0000 - val_tn0.9: 20839018.0000 - val_fn0.9: 1647273.0000 - val_precision0.9: 0.6923 - val_recall0.9: 0.6104 - val_accuracy: 0.8842 - val_auc: 0.9226
  Epoch 21/50
  1280/1280 [==============================] - 2204s 2s/step - loss: 0.2384 - tp0.1: 8140617.7596 - fp0.1: 3066144.5098 - tn0.1: 40972600.1975 - fn0.1: 331283.3044 - precision0.1: 0.7253 - recall0.1: 0.9608 - tp0.3: 7870474.5457 - fp0.3:
   1553244.6440 - tn0.3: 42485524.1530 - fn0.3: 601426.5183 - precision0.3: 0.8351 - recall0.3: 0.9289 - tp0.5: 7506928.4785 - fp0.5: 777322.4692 - tn0.5: 43261445.7369 - fn0.5: 964972.5855 - precision0.5: 0.9063 - recall0.5: 0.8856 - tp
  0.7: 7045491.5386 - fp0.7: 357699.5441 - tn0.7: 43681060.9094 - fn0.7: 1426409.5254 - precision0.7: 0.9516 - recall0.7: 0.8307 - tp0.9: 6309047.4247 - fp0.9: 104611.4294 - tn0.9: 43934120.4317 - fn0.9: 2162853.6393 - precision0.9: 0.98
  35 - recall0.9: 0.7429 - accuracy: 0.9668 - auc: 0.9764 - val_loss: 0.5731 - val_tp0.1: 3942096.0000 - val_fp0.1: 2995461.0000 - val_tn0.1: 18887014.0000 - val_fn0.1: 389824.0000 - val_precision0.1: 0.5682 - val_recall0.1: 0.9100 - val
  _tp0.3: 3599028.0000 - val_fp0.3: 2149008.0000 - val_tn0.3: 19733476.0000 - val_fn0.3: 732892.0000 - val_precision0.3: 0.6261 - val_recall0.3: 0.8308 - val_tp0.5: 3252555.0000 - val_fp0.5: 1599632.0000 - val_tn0.5: 20282852.0000 - val_
  fn0.5: 1079365.0000 - val_precision0.5: 0.6703 - val_recall0.5: 0.7508 - val_tp0.7: 2923628.0000 - val_fp0.7: 1191800.0000 - val_tn0.7: 20690680.0000 - val_fn0.7: 1408292.0000 - val_precision0.7: 0.7104 - val_recall0.7: 0.6749 - val_tp
  0.9: 2521604.0000 - val_fp0.9: 728892.0000 - val_tn0.9: 21153588.0000 - val_fn0.9: 1810316.0000 - val_precision0.9: 0.7758 - val_recall0.9: 0.5821 - val_accuracy: 0.8978 - val_auc: 0.9225
  Epoch 22/50
  1280/1280 [==============================] - 2202s 2s/step - loss: 0.2325 - tp0.1: 8142655.7533 - fp0.1: 2988317.6230 - tn0.1: 41059668.6183 - fn0.1: 320016.3560 - precision0.1: 0.7319 - recall0.1: 0.9621 - tp0.3: 7871852.7736 - fp0.3:
   1506270.7799 - tn0.3: 42541714.4848 - fn0.3: 590819.3357 - precision0.3: 0.8406 - recall0.3: 0.9295 - tp0.5: 7524881.3895 - fp0.5: 772793.9836 - tn0.5: 43275195.0945 - fn0.5: 937790.7198 - precision0.5: 0.9075 - recall0.5: 0.8882 - tp
  0.7: 7077438.3216 - fp0.7: 364340.6877 - tn0.7: 43683658.4301 - fn0.7: 1385233.7877 - precision0.7: 0.9513 - recall0.7: 0.8352 - tp0.9: 6339229.0827 - fp0.9: 109004.6729 - tn0.9: 43938973.3130 - fn0.9: 2123443.0265 - precision0.9: 0.98
  30 - recall0.9: 0.7477 - accuracy: 0.9672 - auc: 0.9774 - val_loss: 0.5088 - val_tp0.1: 4052314.0000 - val_fp0.1: 2813624.0000 - val_tn0.1: 18998480.0000 - val_fn0.1: 349977.0000 - val_precision0.1: 0.5902 - val_recall0.1: 0.9205 - val
  _tp0.3: 3687682.0000 - val_fp0.3: 1991405.0000 - val_tn0.3: 19820706.0000 - val_fn0.3: 714609.0000 - val_precision0.3: 0.6493 - val_recall0.3: 0.8377 - val_tp0.5: 3320706.0000 - val_fp0.5: 1380622.0000 - val_tn0.5: 20431490.0000 - val_
  fn0.5: 1081585.0000 - val_precision0.5: 0.7063 - val_recall0.5: 0.7543 - val_tp0.7: 2967625.0000 - val_fp0.7: 953933.0000 - val_tn0.7: 20858176.0000 - val_fn0.7: 1434666.0000 - val_precision0.7: 0.7567 - val_recall0.7: 0.6741 - val_tp0
  .9: 2531957.0000 - val_fp0.9: 497718.0000 - val_tn0.9: 21314390.0000 - val_fn0.9: 1870334.0000 - val_precision0.9: 0.8357 - val_recall0.9: 0.5751 - val_accuracy: 0.9061 - val_auc: 0.9328
  Epoch 23/50
  1280/1280 [==============================] - 2222s 2s/step - loss: 0.2366 - tp0.1: 8347653.8970 - fp0.1: 3081670.7681 - tn0.1: 40757305.4309 - fn0.1: 324014.3521 - precision0.1: 0.7290 - recall0.1: 0.9623 - tp0.3: 8070883.3950 - fp0.3:
   1575587.0781 - tn0.3: 42263394.8119 - fn0.3: 600784.8540 - precision0.3: 0.8352 - recall0.3: 0.9305 - tp0.5: 7717891.5347 - fp0.5: 814813.8595 - tn0.5: 43024150.6456 - fn0.5: 953776.7143 - precision0.5: 0.9036 - recall0.5: 0.8895 - tp
  0.7: 7268612.4356 - fp0.7: 391109.7408 - tn0.7: 43447878.6557 - fn0.7: 1403055.8134 - precision0.7: 0.9483 - recall0.7: 0.8378 - tp0.9: 6489543.3856 - fp0.9: 111622.6698 - tn0.9: 43727379.5621 - fn0.9: 2182124.8634 - precision0.9: 0.98
  31 - recall0.9: 0.7476 - accuracy: 0.9663 - auc: 0.9772 - val_loss: 0.6052 - val_tp0.1: 3963049.0000 - val_fp0.1: 3225080.0000 - val_tn0.1: 18705446.0000 - val_fn0.1: 320823.0000 - val_precision0.1: 0.5513 - val_recall0.1: 0.9251 - val
  _tp0.3: 3674699.0000 - val_fp0.3: 2561181.0000 - val_tn0.3: 19369346.0000 - val_fn0.3: 609173.0000 - val_precision0.3: 0.5893 - val_recall0.3: 0.8578 - val_tp0.5: 3373791.0000 - val_fp0.5: 2127280.0000 - val_tn0.5: 19803252.0000 - val_
  fn0.5: 910081.0000 - val_precision0.5: 0.6133 - val_recall0.5: 0.7876 - val_tp0.7: 3068979.0000 - val_fp0.7: 1731571.0000 - val_tn0.7: 20198960.0000 - val_fn0.7: 1214893.0000 - val_precision0.7: 0.6393 - val_recall0.7: 0.7164 - val_tp0
  .9: 2716240.0000 - val_fp0.9: 1223118.0000 - val_tn0.9: 20707410.0000 - val_fn0.9: 1567632.0000 - val_precision0.9: 0.6895 - val_recall0.9: 0.6341 - val_accuracy: 0.8841 - val_auc: 0.9237
  Epoch 24/50
  1280/1280 [==============================] - 2204s 2s/step - loss: 0.2289 - tp0.1: 8179279.4356 - fp0.1: 3005191.0726 - tn0.1: 41009950.9407 - fn0.1: 316225.4254 - precision0.1: 0.7325 - recall0.1: 0.9631 - tp0.3: 7901240.7315 - fp0.3:
   1504773.7026 - tn0.3: 42510372.7838 - fn0.3: 594264.1296 - precision0.3: 0.8407 - recall0.3: 0.9307 - tp0.5: 7554121.0180 - fp0.5: 777848.4996 - tn0.5: 43237301.8970 - fn0.5: 941383.8431 - precision0.5: 0.9069 - recall0.5: 0.8901 - tp
  0.7: 7115199.6105 - fp0.7: 370807.8002 - tn0.7: 43644327.6066 - fn0.7: 1380305.2506 - precision0.7: 0.9505 - recall0.7: 0.8385 - tp0.9: 6375731.2256 - fp0.9: 109566.7822 - tn0.9: 43905607.9274 - fn0.9: 2119773.6354 - precision0.9: 0.98
  30 - recall0.9: 0.7519 - accuracy: 0.9674 - auc: 0.9780 - val_loss: 0.5914 - val_tp0.1: 4024334.0000 - val_fp0.1: 3404652.0000 - val_tn0.1: 18506274.0000 - val_fn0.1: 279143.0000 - val_precision0.1: 0.5417 - val_recall0.1: 0.9351 - val
  _tp0.3: 3754317.0000 - val_fp0.3: 2715350.0000 - val_tn0.3: 19195568.0000 - val_fn0.3: 549160.0000 - val_precision0.3: 0.5803 - val_recall0.3: 0.8724 - val_tp0.5: 3470133.0000 - val_fp0.5: 2252806.0000 - val_tn0.5: 19658116.0000 - val_
  fn0.5: 833344.0000 - val_precision0.5: 0.6064 - val_recall0.5: 0.8064 - val_tp0.7: 3161773.0000 - val_fp0.7: 1777262.0000 - val_tn0.7: 20133664.0000 - val_fn0.7: 1141704.0000 - val_precision0.7: 0.6402 - val_recall0.7: 0.7347 - val_tp0
  .9: 2787618.0000 - val_fp0.9: 1197991.0000 - val_tn0.9: 20712926.0000 - val_fn0.9: 1515859.0000 - val_precision0.9: 0.6994 - val_recall0.9: 0.6478 - val_accuracy: 0.8823 - val_auc: 0.9283
  Epoch 25/50
  1280/1280 [==============================] - 2226s 2s/step - loss: 0.2252 - tp0.1: 8291979.4137 - fp0.1: 2984606.2982 - tn0.1: 40931178.3692 - fn0.1: 302890.0656 - precision0.1: 0.7344 - recall0.1: 0.9653 - tp0.3: 8019589.6370 - fp0.3:
   1494741.8806 - tn0.3: 42421057.4286 - fn0.3: 575279.8423 - precision0.3: 0.8422 - recall0.3: 0.9335 - tp0.5: 7686923.1866 - fp0.5: 778234.4176 - tn0.5: 43137535.1632 - fn0.5: 907946.2927 - precision0.5: 0.9079 - recall0.5: 0.8947 - tp
  0.7: 7250107.4707 - fp0.7: 372459.3240 - tn0.7: 43543354.1975 - fn0.7: 1344762.0086 - precision0.7: 0.9509 - recall0.7: 0.8437 - tp0.9: 6493233.8290 - fp0.9: 107705.5613 - tn0.9: 43808088.1733 - fn0.9: 2101635.6503 - precision0.9: 0.98
  38 - recall0.9: 0.7548 - accuracy: 0.9677 - auc: 0.9792 - val_loss: 0.5570 - val_tp0.1: 3995650.0000 - val_fp0.1: 3060009.0000 - val_tn0.1: 18836932.0000 - val_fn0.1: 321813.0000 - val_precision0.1: 0.5663 - val_recall0.1: 0.9255 - val
  _tp0.3: 3674821.0000 - val_fp0.3: 2374036.0000 - val_tn0.3: 19522900.0000 - val_fn0.3: 642642.0000 - val_precision0.3: 0.6075 - val_recall0.3: 0.8512 - val_tp0.5: 3346516.0000 - val_fp0.5: 1900974.0000 - val_tn0.5: 19995958.0000 - val_
  fn0.5: 970947.0000 - val_precision0.5: 0.6377 - val_recall0.5: 0.7751 - val_tp0.7: 3018485.0000 - val_fp0.7: 1460209.0000 - val_tn0.7: 20436728.0000 - val_fn0.7: 1298978.0000 - val_precision0.7: 0.6740 - val_recall0.7: 0.6991 - val_tp0
  .9: 2609054.0000 - val_fp0.9: 880392.0000 - val_tn0.9: 21016544.0000 - val_fn0.9: 1708409.0000 - val_precision0.9: 0.7477 - val_recall0.9: 0.6043 - val_accuracy: 0.8904 - val_auc: 0.9280
  Epoch 26/50
  1280/1280 [==============================] - 2216s 2s/step - loss: 0.2287 - tp0.1: 8319398.9969 - fp0.1: 2952438.8306 - tn0.1: 40921029.8501 - fn0.1: 317787.0890 - precision0.1: 0.7394 - recall0.1: 0.9635 - tp0.3: 8049496.8860 - fp0.3:
   1491537.2982 - tn0.3: 42381938.1233 - fn0.3: 587689.1998 - precision0.3: 0.8439 - recall0.3: 0.9326 - tp0.5: 7714917.0734 - fp0.5: 782715.8696 - tn0.5: 43090766.7041 - fn0.5: 922269.0125 - precision0.5: 0.9077 - recall0.5: 0.8941 - tp
  0.7: 7279420.8743 - fp0.7: 371072.6573 - tn0.7: 43502429.9672 - fn0.7: 1357765.2116 - precision0.7: 0.9513 - recall0.7: 0.8438 - tp0.9: 6527109.8852 - fp0.9: 103689.7635 - tn0.9: 43769772.2685 - fn0.9: 2110076.2006 - precision0.9: 0.98
  45 - recall0.9: 0.7564 - accuracy: 0.9674 - auc: 0.9783 - val_loss: 0.5787 - val_tp0.1: 3910190.0000 - val_fp0.1: 3250030.0000 - val_tn0.1: 18794728.0000 - val_fn0.1: 259453.0000 - val_precision0.1: 0.5461 - val_recall0.1: 0.9378 - val
  _tp0.3: 3639624.0000 - val_fp0.3: 2570780.0000 - val_tn0.3: 19473980.0000 - val_fn0.3: 530019.0000 - val_precision0.3: 0.5861 - val_recall0.3: 0.8729 - val_tp0.5: 3361012.0000 - val_fp0.5: 2146148.0000 - val_tn0.5: 19898600.0000 - val_
  fn0.5: 808631.0000 - val_precision0.5: 0.6103 - val_recall0.5: 0.8061 - val_tp0.7: 3051142.0000 - val_fp0.7: 1790049.0000 - val_tn0.7: 20254708.0000 - val_fn0.7: 1118501.0000 - val_precision0.7: 0.6302 - val_recall0.7: 0.7318 - val_tp0
  .9: 2683794.0000 - val_fp0.9: 1283960.0000 - val_tn0.9: 20760796.0000 - val_fn0.9: 1485849.0000 - val_precision0.9: 0.6764 - val_recall0.9: 0.6437 - val_accuracy: 0.8873 - val_auc: 0.9297
  Epoch 27/50
  1280/1280 [==============================] - 2202s 2s/step - loss: 0.2301 - tp0.1: 8107973.9399 - fp0.1: 2902140.6105 - tn0.1: 41182030.2319 - fn0.1: 318498.9274 - precision0.1: 0.7370 - recall0.1: 0.9620 - tp0.3: 7841754.6401 - fp0.3:
   1478702.1952 - tn0.3: 42605477.3365 - fn0.3: 584718.2272 - precision0.3: 0.8414 - recall0.3: 0.9300 - tp0.5: 7511893.1749 - fp0.5: 775511.8610 - tn0.5: 43308683.9024 - fn0.5: 914579.6924 - precision0.5: 0.9061 - recall0.5: 0.8905 - tp
  0.7: 7075173.0062 - fp0.7: 368300.9977 - tn0.7: 43715881.6714 - fn0.7: 1351299.8610 - precision0.7: 0.9500 - recall0.7: 0.8385 - tp0.9: 6326135.5004 - fp0.9: 105903.6909 - tn0.9: 43978285.1374 - fn0.9: 2100337.3669 - precision0.9: 0.98
  29 - recall0.9: 0.7495 - accuracy: 0.9673 - auc: 0.9774 - val_loss: 0.5563 - val_tp0.1: 3973953.0000 - val_fp0.1: 3033629.0000 - val_tn0.1: 18923208.0000 - val_fn0.1: 283603.0000 - val_precision0.1: 0.5671 - val_recall0.1: 0.9334 - val
  _tp0.3: 3702270.0000 - val_fp0.3: 2375708.0000 - val_tn0.3: 19581140.0000 - val_fn0.3: 555286.0000 - val_precision0.3: 0.6091 - val_recall0.3: 0.8696 - val_tp0.5: 3427641.0000 - val_fp0.5: 1972155.0000 - val_tn0.5: 19984684.0000 - val_
  fn0.5: 829915.0000 - val_precision0.5: 0.6348 - val_recall0.5: 0.8051 - val_tp0.7: 3124472.0000 - val_fp0.7: 1581643.0000 - val_tn0.7: 20375196.0000 - val_fn0.7: 1133084.0000 - val_precision0.7: 0.6639 - val_recall0.7: 0.7339 - val_tp0
  .9: 2757974.0000 - val_fp0.9: 1109127.0000 - val_tn0.9: 20847720.0000 - val_fn0.9: 1499582.0000 - val_precision0.9: 0.7132 - val_recall0.9: 0.6478 - val_accuracy: 0.8931 - val_auc: 0.9321
  Epoch 28/50
  1280/1280 [==============================] - 2087s 2s/step - loss: 0.2197 - tp0.1: 8160795.8228 - fp0.1: 2901034.4309 - tn0.1: 41142814.8181 - fn0.1: 306021.3466 - precision0.1: 0.7381 - recall0.1: 0.9644 - tp0.3: 7892365.4137 - fp0.3:
   1440828.1148 - tn0.3: 42603028.9719 - fn0.3: 574451.7557 - precision0.3: 0.8461 - recall0.3: 0.9326 - tp0.5: 7566706.5480 - fp0.5: 755226.4699 - tn0.5: 43288602.1678 - fn0.5: 900110.6214 - precision0.5: 0.9096 - recall0.5: 0.8942 - tp
  0.7: 7138328.0726 - fp0.7: 359169.3607 - tn0.7: 43684677.2763 - fn0.7: 1328489.0968 - precision0.7: 0.9523 - recall0.7: 0.8435 - tp0.9: 6407064.9711 - fp0.9: 100987.8735 - tn0.9: 43942848.2186 - fn0.9: 2059752.1983 - precision0.9: 0.98
  47 - recall0.9: 0.7566 - accuracy: 0.9687 - auc: 0.9791 - val_loss: 0.4500 - val_tp0.1: 3927337.0000 - val_fp0.1: 2709904.0000 - val_tn0.1: 19279502.0000 - val_fn0.1: 297660.0000 - val_precision0.1: 0.5917 - val_recall0.1: 0.9295 - val
  _tp0.3: 3622327.0000 - val_fp0.3: 1934449.0000 - val_tn0.3: 20054960.0000 - val_fn0.3: 602670.0000 - val_precision0.3: 0.6519 - val_recall0.3: 0.8574 - val_tp0.5: 3304686.0000 - val_fp0.5: 1267577.0000 - val_tn0.5: 20721824.0000 - val_
  fn0.5: 920311.0000 - val_precision0.5: 0.7228 - val_recall0.5: 0.7822 - val_tp0.7: 2969971.0000 - val_fp0.7: 803372.0000 - val_tn0.7: 21186032.0000 - val_fn0.7: 1255026.0000 - val_precision0.7: 0.7871 - val_recall0.7: 0.7030 - val_tp0.
  9: 2545450.0000 - val_fp0.9: 373527.0000 - val_tn0.9: 21615866.0000 - val_fn0.9: 1679547.0000 - val_precision0.9: 0.8720 - val_recall0.9: 0.6025 - val_accuracy: 0.9165 - val_auc: 0.9426
  Epoch 29/50
  1280/1280 [==============================] - 1968s 2s/step - loss: 0.2223 - tp0.1: 8375369.2490 - fp0.1: 2828068.0578 - tn0.1: 41000432.9828 - fn0.1: 306798.1413 - precision0.1: 0.7500 - recall0.1: 0.9648 - tp0.3: 8117816.3411 - fp0.3:
   1454670.2927 - tn0.3: 42373821.8306 - fn0.3: 564351.0492 - precision0.3: 0.8489 - recall0.3: 0.9354 - tp0.5: 7795654.3497 - fp0.5: 777398.3349 - tn0.5: 43051103.4832 - fn0.5: 886513.0406 - precision0.5: 0.9098 - recall0.5: 0.8987 - tp
  0.7: 7368227.7635 - fp0.7: 376034.1069 - tn0.7: 43452466.5371 - fn0.7: 1313939.6269 - precision0.7: 0.9517 - recall0.7: 0.8498 - tp0.9: 6617754.3536 - fp0.9: 111597.3193 - tn0.9: 43716853.5761 - fn0.9: 2064413.0367 - precision0.9: 0.98
  35 - recall0.9: 0.7637 - accuracy: 0.9681 - auc: 0.9792 - val_loss: 0.5416 - val_tp0.1: 3960357.0000 - val_fp0.1: 3230144.0000 - val_tn0.1: 18775996.0000 - val_fn0.1: 247899.0000 - val_precision0.1: 0.5508 - val_recall0.1: 0.9411 - val
  _tp0.3: 3709415.0000 - val_fp0.3: 2490342.0000 - val_tn0.3: 19515800.0000 - val_fn0.3: 498841.0000 - val_precision0.3: 0.5983 - val_recall0.3: 0.8815 - val_tp0.5: 3451033.0000 - val_fp0.5: 2038241.0000 - val_tn0.5: 19967904.0000 - val_
  fn0.5: 757223.0000 - val_precision0.5: 0.6287 - val_recall0.5: 0.8201 - val_tp0.7: 3148179.0000 - val_fp0.7: 1621294.0000 - val_tn0.7: 20384850.0000 - val_fn0.7: 1060077.0000 - val_precision0.7: 0.6601 - val_recall0.7: 0.7481 - val_tp0
  .9: 2788544.0000 - val_fp0.9: 1101598.0000 - val_tn0.9: 20904542.0000 - val_fn0.9: 1419712.0000 - val_precision0.9: 0.7168 - val_recall0.9: 0.6626 - val_accuracy: 0.8934 - val_auc: 0.9369
  Epoch 30/50
  1280/1280 [==============================] - 1987s 2s/step - loss: 0.2171 - tp0.1: 8237924.9360 - fp0.1: 2817507.8111 - tn0.1: 41152156.1608 - fn0.1: 303072.4856 - precision0.1: 0.7480 - recall0.1: 0.9644 - tp0.3: 7981439.7697 - fp0.3:
   1425495.2022 - tn0.3: 42544167.9969 - fn0.3: 559557.6518 - precision0.3: 0.8500 - recall0.3: 0.9347 - tp0.5: 7663120.4309 - fp0.5: 752759.7705 - tn0.5: 43216900.0492 - fn0.5: 877876.9906 - precision0.5: 0.9116 - recall0.5: 0.8978 - tp
  0.7: 7238382.6721 - fp0.7: 357392.6354 - tn0.7: 43612263.4215 - fn0.7: 1302614.7494 - precision0.7: 0.9538 - recall0.7: 0.8481 - tp0.9: 6495143.9938 - fp0.9: 101203.1015 - tn0.9: 43868469.8891 - fn0.9: 2045853.4278 - precision0.9: 0.98
  52 - recall0.9: 0.7605 - accuracy: 0.9692 - auc: 0.9793 - val_loss: 0.5477 - val_tp0.1: 4026668.0000 - val_fp0.1: 3118480.0000 - val_tn0.1: 18786898.0000 - val_fn0.1: 282350.0000 - val_precision0.1: 0.5636 - val_recall0.1: 0.9345 - val
  _tp0.3: 3769431.0000 - val_fp0.3: 2411082.0000 - val_tn0.3: 19494304.0000 - val_fn0.3: 539587.0000 - val_precision0.3: 0.6099 - val_recall0.3: 0.8748 - val_tp0.5: 3496580.0000 - val_fp0.5: 1951231.0000 - val_tn0.5: 19954152.0000 - val_
  fn0.5: 812438.0000 - val_precision0.5: 0.6418 - val_recall0.5: 0.8115 - val_tp0.7: 3175729.0000 - val_fp0.7: 1538001.0000 - val_tn0.7: 20367384.0000 - val_fn0.7: 1133289.0000 - val_precision0.7: 0.6737 - val_recall0.7: 0.7370 - val_tp0
  .9: 2769656.0000 - val_fp0.9: 1014413.0000 - val_tn0.9: 20890968.0000 - val_fn0.9: 1539362.0000 - val_precision0.9: 0.7319 - val_recall0.9: 0.6428 - val_accuracy: 0.8946 - val_auc: 0.9332
  Epoch 31/50
  1280/1280 [==============================] - 1975s 2s/step - loss: 0.2140 - tp0.1: 8254893.9524 - fp0.1: 2820541.8767 - tn0.1: 41143836.3778 - fn0.1: 291401.7947 - precision0.1: 0.7449 - recall0.1: 0.9663 - tp0.3: 8001833.3880 - fp0.3:
   1415982.6237 - tn0.3: 42548340.3044 - fn0.3: 544462.3591 - precision0.3: 0.8490 - recall0.3: 0.9370 - tp0.5: 7689740.5105 - fp0.5: 749876.5176 - tn0.5: 43214484.3458 - fn0.5: 856555.2365 - precision0.5: 0.9104 - recall0.5: 0.9008 - tp
  0.7: 7273005.2974 - fp0.7: 357730.3216 - tn0.7: 43606618.3302 - fn0.7: 1273290.4496 - precision0.7: 0.9525 - recall0.7: 0.8527 - tp0.9: 6528857.1772 - fp0.9: 101215.2264 - tn0.9: 43863149.4848 - fn0.9: 2017438.5699 - precision0.9: 0.98
  45 - recall0.9: 0.7661 - accuracy: 0.9694 - auc: 0.9801 - val_loss: 0.6223 - val_tp0.1: 3930460.0000 - val_fp0.1: 3394464.0000 - val_tn0.1: 18612148.0000 - val_fn0.1: 277327.0000 - val_precision0.1: 0.5366 - val_recall0.1: 0.9341 - val
  _tp0.3: 3651274.0000 - val_fp0.3: 2652264.0000 - val_tn0.3: 19354352.0000 - val_fn0.3: 556513.0000 - val_precision0.3: 0.5792 - val_recall0.3: 0.8677 - val_tp0.5: 3363770.0000 - val_fp0.5: 2181393.0000 - val_tn0.5: 19825220.0000 - val_
  fn0.5: 844017.0000 - val_precision0.5: 0.6066 - val_recall0.5: 0.7994 - val_tp0.7: 3042185.0000 - val_fp0.7: 1805943.0000 - val_tn0.7: 20200670.0000 - val_fn0.7: 1165602.0000 - val_precision0.7: 0.6275 - val_recall0.7: 0.7230 - val_tp0
  .9: 2649971.0000 - val_fp0.9: 1376978.0000 - val_tn0.9: 20629630.0000 - val_fn0.9: 1557816.0000 - val_precision0.9: 0.6581 - val_recall0.9: 0.6298 - val_accuracy: 0.8846 - val_auc: 0.9230
  Epoch 32/50
  1280/1280 [==============================] - 1985s 2s/step - loss: 0.2162 - tp0.1: 8168329.4106 - fp0.1: 2823874.2186 - tn0.1: 41219292.6159 - fn0.1: 299147.5254 - precision0.1: 0.7416 - recall0.1: 0.9648 - tp0.3: 7902494.5543 - fp0.3:
   1406775.2030 - tn0.3: 42636384.1311 - fn0.3: 564982.3817 - precision0.3: 0.8486 - recall0.3: 0.9329 - tp0.5: 7587653.1304 - fp0.5: 746883.7400 - tn0.5: 43296303.4005 - fn0.5: 879823.8056 - precision0.5: 0.9100 - recall0.5: 0.8957 - tp
  0.7: 7172023.5183 - fp0.7: 358490.8813 - tn0.7: 43684701.1390 - fn0.7: 1295453.4176 - precision0.7: 0.9519 - recall0.7: 0.8466 - tp0.9: 6440127.4231 - fp0.9: 99619.9977 - tn0.9: 43943567.7978 - fn0.9: 2027349.5129 - precision0.9: 0.984
  4 - recall0.9: 0.7605 - accuracy: 0.9691 - auc: 0.9792 - val_loss: 0.5922 - val_tp0.1: 4021661.0000 - val_fp0.1: 3341968.0000 - val_tn0.1: 18581152.0000 - val_fn0.1: 269622.0000 - val_precision0.1: 0.5462 - val_recall0.1: 0.9372 - val_
  tp0.3: 3723109.0000 - val_fp0.3: 2641493.0000 - val_tn0.3: 19281632.0000 - val_fn0.3: 568174.0000 - val_precision0.3: 0.5850 - val_recall0.3: 0.8676 - val_tp0.5: 3417861.0000 - val_fp0.5: 2185964.0000 - val_tn0.5: 19737154.0000 - val_f
  n0.5: 873422.0000 - val_precision0.5: 0.6099 - val_recall0.5: 0.7965 - val_tp0.7: 3082813.0000 - val_fp0.7: 1779173.0000 - val_tn0.7: 20143952.0000 - val_fn0.7: 1208470.0000 - val_precision0.7: 0.6341 - val_recall0.7: 0.7184 - val_tp0.
  9: 2687718.0000 - val_fp0.9: 1223941.0000 - val_tn0.9: 20699176.0000 - val_fn0.9: 1603565.0000 - val_precision0.9: 0.6871 - val_recall0.9: 0.6263 - val_accuracy: 0.8833 - val_auc: 0.9267
  Epoch 33/50
  1280/1280 [==============================] - 1990s 2s/step - loss: 0.2153 - tp0.1: 8159566.2951 - fp0.1: 2769324.2404 - tn0.1: 41293176.3880 - fn0.1: 288608.7385 - precision0.1: 0.7434 - recall0.1: 0.9660 - tp0.3: 7912034.2069 - fp0.3:
   1418334.3794 - tn0.3: 42644140.5886 - fn0.3: 536140.8267 - precision0.3: 0.8451 - recall0.3: 0.9365 - tp0.5: 7606237.6534 - fp0.5: 760974.3443 - tn0.5: 43301518.3575 - fn0.5: 841937.3802 - precision0.5: 0.9067 - recall0.5: 0.9000 - tp
  0.7: 7188970.2584 - fp0.7: 368072.7486 - tn0.7: 43694407.5129 - fn0.7: 1259204.7752 - precision0.7: 0.9494 - recall0.7: 0.8501 - tp0.9: 6447265.7268 - fp0.9: 106096.6183 - tn0.9: 43956391.5020 - fn0.9: 2000909.3068 - precision0.9: 0.98
  25 - recall0.9: 0.7622 - accuracy: 0.9690 - auc: 0.9797 - val_loss: 0.6103 - val_tp0.1: 4142693.0000 - val_fp0.1: 3021502.0000 - val_tn0.1: 18751128.0000 - val_fn0.1: 299079.0000 - val_precision0.1: 0.5782 - val_recall0.1: 0.9327 - val
  _tp0.3: 3873213.0000 - val_fp0.3: 2448264.0000 - val_tn0.3: 19324360.0000 - val_fn0.3: 568559.0000 - val_precision0.3: 0.6127 - val_recall0.3: 0.8720 - val_tp0.5: 3610020.0000 - val_fp0.5: 2106610.0000 - val_tn0.5: 19666014.0000 - val_
  fn0.5: 831752.0000 - val_precision0.5: 0.6315 - val_recall0.5: 0.8127 - val_tp0.7: 3302152.0000 - val_fp0.7: 1796893.0000 - val_tn0.7: 19975734.0000 - val_fn0.7: 1139620.0000 - val_precision0.7: 0.6476 - val_recall0.7: 0.7434 - val_tp0
  .9: 2940645.0000 - val_fp0.9: 1387374.0000 - val_tn0.9: 20385260.0000 - val_fn0.9: 1501127.0000 - val_precision0.9: 0.6794 - val_recall0.9: 0.6620 - val_accuracy: 0.8879 - val_auc: 0.9267
  Epoch 34/50
  1280/1280 [==============================] - 1975s 2s/step - loss: 0.2140 - tp0.1: 8354952.6027 - fp0.1: 2805323.7728 - tn0.1: 41057643.9641 - fn0.1: 292731.8275 - precision0.1: 0.7496 - recall0.1: 0.9661 - tp0.3: 8100316.4520 - fp0.3:
   1408795.9087 - tn0.3: 42454187.0320 - fn0.3: 547367.9781 - precision0.3: 0.8514 - recall0.3: 0.9370 - tp0.5: 7794247.9141 - fp0.5: 754289.6760 - tn0.5: 43108644.9532 - fn0.5: 853436.5160 - precision0.5: 0.9115 - recall0.5: 0.9017 - tp
  0.7: 7381753.1905 - fp0.7: 366912.4387 - tn0.7: 43496065.3107 - fn0.7: 1265931.2397 - precision0.7: 0.9524 - recall0.7: 0.8539 - tp0.9: 6643385.3435 - fp0.9: 104367.8337 - tn0.9: 43758580.2849 - fn0.9: 2004299.0867 - precision0.9: 0.98
  45 - recall0.9: 0.7683 - accuracy: 0.9694 - auc: 0.9801 - val_loss: 0.6624 - val_tp0.1: 3856354.0000 - val_fp0.1: 3145321.0000 - val_tn0.1: 18842388.0000 - val_fn0.1: 370344.0000 - val_precision0.1: 0.5508 - val_recall0.1: 0.9124 - val
  _tp0.3: 3549372.0000 - val_fp0.3: 2542341.0000 - val_tn0.3: 19445350.0000 - val_fn0.3: 677326.0000 - val_precision0.3: 0.5827 - val_recall0.3: 0.8398 - val_tp0.5: 3241294.0000 - val_fp0.5: 2119984.0000 - val_tn0.5: 19867720.0000 - val_
  fn0.5: 985404.0000 - val_precision0.5: 0.6046 - val_recall0.5: 0.7669 - val_tp0.7: 2930029.0000 - val_fp0.7: 1781338.0000 - val_tn0.7: 20206368.0000 - val_fn0.7: 1296669.0000 - val_precision0.7: 0.6219 - val_recall0.7: 0.6932 - val_tp0
  .9: 2550993.0000 - val_fp0.9: 1347073.0000 - val_tn0.9: 20640632.0000 - val_fn0.9: 1675705.0000 - val_precision0.9: 0.6544 - val_recall0.9: 0.6035 - val_accuracy: 0.8815 - val_auc: 0.9118
  Epoch 35/50
  1280/1280 [==============================] - 1979s 2s/step - loss: 0.2136 - tp0.1: 8121225.3489 - fp0.1: 2767519.3271 - tn0.1: 41328977.4340 - fn0.1: 292944.1897 - precision0.1: 0.7441 - recall0.1: 0.9648 - tp0.3: 7867775.7931 - fp0.3:
   1380664.7354 - tn0.3: 42715785.1093 - fn0.3: 546393.7455 - precision0.3: 0.8496 - recall0.3: 0.9344 - tp0.5: 7568656.2834 - fp0.5: 741064.5824 - tn0.5: 43355403.8126 - fn0.5: 845513.2553 - precision0.5: 0.9099 - recall0.5: 0.8989 - tp
  0.7: 7163273.2842 - fp0.7: 355917.4442 - tn0.7: 43740534.9688 - fn0.7: 1250896.2545 - precision0.7: 0.9521 - recall0.7: 0.8507 - tp0.9: 6427745.3263 - fp0.9: 99399.9547 - tn0.9: 43997102.2342 - fn0.9: 1986424.2123 - precision0.9: 0.984
  3 - recall0.9: 0.7629 - accuracy: 0.9697 - auc: 0.9794 - val_loss: 0.6109 - val_tp0.1: 3805391.0000 - val_fp0.1: 3155702.0000 - val_tn0.1: 18859740.0000 - val_fn0.1: 393569.0000 - val_precision0.1: 0.5467 - val_recall0.1: 0.9063 - val_
  tp0.3: 3505616.0000 - val_fp0.3: 2482634.0000 - val_tn0.3: 19532808.0000 - val_fn0.3: 693344.0000 - val_precision0.3: 0.5854 - val_recall0.3: 0.8349 - val_tp0.5: 3205095.0000 - val_fp0.5: 1968560.0000 - val_tn0.5: 20046874.0000 - val_f
  n0.5: 993865.0000 - val_precision0.5: 0.6195 - val_recall0.5: 0.7633 - val_tp0.7: 2889262.0000 - val_fp0.7: 1535981.0000 - val_tn0.7: 20479460.0000 - val_fn0.7: 1309698.0000 - val_precision0.7: 0.6529 - val_recall0.7: 0.6881 - val_tp0.
  9: 2477042.0000 - val_fp0.9: 982485.0000 - val_tn0.9: 21032952.0000 - val_fn0.9: 1721918.0000 - val_precision0.9: 0.7160 - val_recall0.9: 0.5899 - val_accuracy: 0.8870 - val_auc: 0.9160
  Epoch 36/50
  1280/1280 [==============================] - 1979s 2s/step - loss: 0.2059 - tp0.1: 8126280.0625 - fp0.1: 2646687.9227 - tn0.1: 41454888.0211 - fn0.1: 282807.8095 - precision0.1: 0.7538 - recall0.1: 0.9656 - tp0.3: 7878883.7135 - fp0.3:
   1323333.3552 - tn0.3: 42778233.2834 - fn0.3: 530204.1585 - precision0.3: 0.8559 - recall0.3: 0.9362 - tp0.5: 7591944.3318 - fp0.5: 712271.5222 - tn0.5: 43389291.7783 - fn0.5: 817143.5402 - precision0.5: 0.9140 - recall0.5: 0.9022 - tp
  0.7: 7203686.0226 - fp0.7: 343437.4106 - tn0.7: 43758143.1655 - fn0.7: 1205401.8493 - precision0.7: 0.9542 - recall0.7: 0.8564 - tp0.9: 6507343.4036 - fp0.9: 97816.8470 - tn0.9: 44003797.8345 - fn0.9: 1901744.4684 - precision0.9: 0.985
  0 - recall0.9: 0.7738 - accuracy: 0.9709 - auc: 0.9801 - val_loss: 0.5870 - val_tp0.1: 4142690.0000 - val_fp0.1: 3187548.0000 - val_tn0.1: 18594374.0000 - val_fn0.1: 289788.0000 - val_precision0.1: 0.5652 - val_recall0.1: 0.9346 - val_
  tp0.3: 3873916.0000 - val_fp0.3: 2590674.0000 - val_tn0.3: 19191250.0000 - val_fn0.3: 558562.0000 - val_precision0.3: 0.5993 - val_recall0.3: 0.8740 - val_tp0.5: 3589951.0000 - val_fp0.5: 2186008.0000 - val_tn0.5: 19595906.0000 - val_f
  n0.5: 842527.0000 - val_precision0.5: 0.6215 - val_recall0.5: 0.8099 - val_tp0.7: 3248583.0000 - val_fp0.7: 1758534.0000 - val_tn0.7: 20023388.0000 - val_fn0.7: 1183895.0000 - val_precision0.7: 0.6488 - val_recall0.7: 0.7329 - val_tp0.
  9: 2813690.0000 - val_fp0.9: 1197675.0000 - val_tn0.9: 20584246.0000 - val_fn0.9: 1618788.0000 - val_precision0.9: 0.7014 - val_recall0.9: 0.6348 - val_accuracy: 0.8845 - val_auc: 0.9275
  Epoch 37/50
  1280/1280 [==============================] - 1993s 2s/step - loss: 0.2130 - tp0.1: 8244281.2553 - fp0.1: 2777623.1632 - tn0.1: 41197305.6495 - fn0.1: 291461.3638 - precision0.1: 0.7465 - recall0.1: 0.9665 - tp0.3: 7992234.6581 - fp0.3:
   1376180.9282 - tn0.3: 42598749.7158 - fn0.3: 543507.9610 - precision0.3: 0.8514 - recall0.3: 0.9375 - tp0.5: 7696124.9321 - fp0.5: 737935.7494 - tn0.5: 43236985.3677 - fn0.5: 839617.6870 - precision0.5: 0.9113 - recall0.5: 0.9031 - tp
  0.7: 7289915.6175 - fp0.7: 354145.8751 - tn0.7: 43620781.5706 - fn0.7: 1245827.0016 - precision0.7: 0.9529 - recall0.7: 0.8559 - tp0.9: 6568139.6253 - fp0.9: 99108.1030 - tn0.9: 43875848.4980 - fn0.9: 1967602.9938 - precision0.9: 0.985
  0 - recall0.9: 0.7707 - accuracy: 0.9697 - auc: 0.9803 - val_loss: 0.6359 - val_tp0.1: 3972718.0000 - val_fp0.1: 2970398.0000 - val_tn0.1: 18895362.0000 - val_fn0.1: 375917.0000 - val_precision0.1: 0.5722 - val_recall0.1: 0.9136 - val_
  tp0.3: 3696968.0000 - val_fp0.3: 2407542.0000 - val_tn0.3: 19458224.0000 - val_fn0.3: 651667.0000 - val_precision0.3: 0.6056 - val_recall0.3: 0.8501 - val_tp0.5: 3418831.0000 - val_fp0.5: 2053387.0000 - val_tn0.5: 19812380.0000 - val_f
  n0.5: 929804.0000 - val_precision0.5: 0.6248 - val_recall0.5: 0.7862 - val_tp0.7: 3101481.0000 - val_fp0.7: 1742891.0000 - val_tn0.7: 20122868.0000 - val_fn0.7: 1247154.0000 - val_precision0.7: 0.6402 - val_recall0.7: 0.7132 - val_tp0.
  9: 2727270.0000 - val_fp0.9: 1316358.0000 - val_tn0.9: 20549408.0000 - val_fn0.9: 1621365.0000 - val_precision0.9: 0.6745 - val_recall0.9: 0.6272 - val_accuracy: 0.8862 - val_auc: 0.9173
  Epoch 38/50
  1280/1280 [==============================] - 1988s 2s/step - loss: 0.2095 - tp0.1: 8322547.4973 - fp0.1: 2736467.1725 - tn0.1: 41169366.0554 - fn0.1: 282238.6792 - precision0.1: 0.7515 - recall0.1: 0.9670 - tp0.3: 8066398.6003 - fp0.3:
   1365148.2873 - tn0.3: 42540740.5316 - fn0.3: 538387.5761 - precision0.3: 0.8549 - recall0.3: 0.9365 - tp0.5: 7770556.2529 - fp0.5: 735978.7931 - tn0.5: 43169885.3575 - fn0.5: 834229.9235 - precision0.5: 0.9132 - recall0.5: 0.9022 - tp
  0.7: 7368501.9875 - fp0.7: 354363.2201 - tn0.7: 43551529.7572 - fn0.7: 1236284.1889 - precision0.7: 0.9541 - recall0.7: 0.8557 - tp0.9: 6640326.0984 - fp0.9: 97963.4192 - tn0.9: 43807909.1975 - fn0.9: 1964460.0781 - precision0.9: 0.985
  5 - recall0.9: 0.7709 - accuracy: 0.9698 - auc: 0.9806 - val_loss: 0.5532 - val_tp0.1: 4070077.0000 - val_fp0.1: 3091837.0000 - val_tn0.1: 18834880.0000 - val_fn0.1: 217612.0000 - val_precision0.1: 0.5683 - val_recall0.1: 0.9492 - val_
  tp0.3: 3841354.0000 - val_fp0.3: 2343330.0000 - val_tn0.3: 19583380.0000 - val_fn0.3: 446335.0000 - val_precision0.3: 0.6211 - val_recall0.3: 0.8959 - val_tp0.5: 3610309.0000 - val_fp0.5: 1940937.0000 - val_tn0.5: 19985782.0000 - val_f
  n0.5: 677380.0000 - val_precision0.5: 0.6504 - val_recall0.5: 0.8420 - val_tp0.7: 3328203.0000 - val_fp0.7: 1620408.0000 - val_tn0.7: 20306288.0000 - val_fn0.7: 959486.0000 - val_precision0.7: 0.6726 - val_recall0.7: 0.7762 - val_tp0.9
  : 2957699.0000 - val_fp0.9: 1252356.0000 - val_tn0.9: 20674356.0000 - val_fn0.9: 1329990.0000 - val_precision0.9: 0.7025 - val_recall0.9: 0.6898 - val_accuracy: 0.9001 - val_auc: 0.9387
  Epoch 39/50
  1280/1280 [==============================] - 1983s 2s/step - loss: 0.2144 - tp0.1: 8366575.3263 - fp0.1: 2766167.9742 - tn0.1: 41080636.4215 - fn0.1: 297288.0242 - precision0.1: 0.7524 - recall0.1: 0.9656 - tp0.3: 8113164.2248 - fp0.3:
   1368911.6565 - tn0.3: 42477877.7752 - fn0.3: 550699.1257 - precision0.3: 0.8555 - recall0.3: 0.9368 - tp0.5: 7813941.1093 - fp0.5: 738235.6479 - tn0.5: 43108568.5925 - fn0.5: 849922.2412 - precision0.5: 0.9133 - recall0.5: 0.9025 - tp
  0.7: 7408230.3068 - fp0.7: 358148.5066 - tn0.7: 43488651.3341 - fn0.7: 1255633.0437 - precision0.7: 0.9536 - recall0.7: 0.8560 - tp0.9: 6668979.4731 - fp0.9: 100667.4668 - tn0.9: 43746134.9883 - fn0.9: 1994883.8774 - precision0.9: 0.98
  51 - recall0.9: 0.7710 - accuracy: 0.9697 - auc: 0.9798 - val_loss: 0.5509 - val_tp0.1: 3928121.0000 - val_fp0.1: 3362312.0000 - val_tn0.1: 18683104.0000 - val_fn0.1: 240862.0000 - val_precision0.1: 0.5388 - val_recall0.1: 0.9422 - val
  _tp0.3: 3683700.0000 - val_fp0.3: 2630754.0000 - val_tn0.3: 19414668.0000 - val_fn0.3: 485283.0000 - val_precision0.3: 0.5834 - val_recall0.3: 0.8836 - val_tp0.5: 3409649.0000 - val_fp0.5: 2123677.0000 - val_tn0.5: 19921740.0000 - val_
  fn0.5: 759334.0000 - val_precision0.5: 0.6162 - val_recall0.5: 0.8179 - val_tp0.7: 3082747.0000 - val_fp0.7: 1673717.0000 - val_tn0.7: 20371692.0000 - val_fn0.7: 1086236.0000 - val_precision0.7: 0.6481 - val_recall0.7: 0.7394 - val_tp0
  .9: 2675367.0000 - val_fp0.9: 1106165.0000 - val_tn0.9: 20939250.0000 - val_fn0.9: 1493616.0000 - val_precision0.9: 0.7075 - val_recall0.9: 0.6417 - val_accuracy: 0.8900 - val_auc: 0.9336
  Epoch 40/50
  1280/1280 [==============================] - 1983s 2s/step - loss: 0.1969 - tp0.1: 8112039.9274 - fp0.1: 2648956.8977 - tn0.1: 41481438.0515 - fn0.1: 268231.5964 - precision0.1: 0.7550 - recall0.1: 0.9682 - tp0.3: 7868925.6425 - fp0.3:
   1315040.1202 - tn0.3: 42815365.2927 - fn0.3: 511345.8813 - precision0.3: 0.8571 - recall0.3: 0.9396 - tp0.5: 7584341.2420 - fp0.5: 709881.4247 - tn0.5: 43420518.9040 - fn0.5: 795930.2818 - precision0.5: 0.9147 - recall0.5: 0.9057 - tp
  0.7: 7193185.0031 - fp0.7: 342578.8923 - tn0.7: 43787824.5191 - fn0.7: 1187086.5207 - precision0.7: 0.9550 - recall0.7: 0.8587 - tp0.9: 6494738.0343 - fp0.9: 97088.4145 - tn0.9: 44033315.7065 - fn0.9: 1885533.4895 - precision0.9: 0.985
  6 - recall0.9: 0.7753 - accuracy: 0.9718 - auc: 0.9815 - val_loss: 0.5690 - val_tp0.1: 3846468.0000 - val_fp0.1: 3007369.0000 - val_tn0.1: 19101806.0000 - val_fn0.1: 258769.0000 - val_precision0.1: 0.5612 - val_recall0.1: 0.9370 - val_
  tp0.3: 3613817.0000 - val_fp0.3: 2397217.0000 - val_tn0.3: 19711954.0000 - val_fn0.3: 491420.0000 - val_precision0.3: 0.6012 - val_recall0.3: 0.8803 - val_tp0.5: 3367535.0000 - val_fp0.5: 2008822.0000 - val_tn0.5: 20100340.0000 - val_f
  n0.5: 737702.0000 - val_precision0.5: 0.6264 - val_recall0.5: 0.8203 - val_tp0.7: 3076985.0000 - val_fp0.7: 1689883.0000 - val_tn0.7: 20419284.0000 - val_fn0.7: 1028252.0000 - val_precision0.7: 0.6455 - val_recall0.7: 0.7495 - val_tp0.
  9: 2719452.0000 - val_fp0.9: 1280160.0000 - val_tn0.9: 20829004.0000 - val_fn0.9: 1385785.0000 - val_precision0.9: 0.6799 - val_recall0.9: 0.6624 - val_accuracy: 0.8952 - val_auc: 0.9311
  Epoch 41/50
  1280/1280 [==============================] - 1978s 2s/step - loss: 0.2044 - tp0.1: 8119761.2217 - fp0.1: 2626596.4067 - tn0.1: 41483268.4879 - fn0.1: 281035.5894 - precision0.1: 0.7571 - recall0.1: 0.9663 - tp0.3: 7878864.4973 - fp0.3:
   1320586.7525 - tn0.3: 42789243.0968 - fn0.3: 521932.3138 - precision0.3: 0.8575 - recall0.3: 0.9377 - tp0.5: 7590439.7557 - fp0.5: 709361.4364 - tn0.5: 43400490.0500 - fn0.5: 810357.0554 - precision0.5: 0.9153 - recall0.5: 0.9035 - tp
  0.7: 7196681.5800 - fp0.7: 343281.3755 - tn0.7: 43766582.6877 - fn0.7: 1204115.2311 - precision0.7: 0.9549 - recall0.7: 0.8569 - tp0.9: 6477320.2436 - fp0.9: 94404.4832 - tn0.9: 44015438.6300 - fn0.9: 1923476.5675 - precision0.9: 0.985
  7 - recall0.9: 0.7709 - accuracy: 0.9710 - auc: 0.9804 - val_loss: 0.7317 - val_tp0.1: 3916023.0000 - val_fp0.1: 3484452.0000 - val_tn0.1: 18565136.0000 - val_fn0.1: 248781.0000 - val_precision0.1: 0.5292 - val_recall0.1: 0.9403 - val_
  tp0.3: 3708159.0000 - val_fp0.3: 2970626.0000 - val_tn0.3: 19078970.0000 - val_fn0.3: 456645.0000 - val_precision0.3: 0.5552 - val_recall0.3: 0.8904 - val_tp0.5: 3475102.0000 - val_fp0.5: 2669322.0000 - val_tn0.5: 19380278.0000 - val_f
  n0.5: 689702.0000 - val_precision0.5: 0.5656 - val_recall0.5: 0.8344 - val_tp0.7: 3188450.0000 - val_fp0.7: 2414665.0000 - val_tn0.7: 19634936.0000 - val_fn0.7: 976354.0000 - val_precision0.7: 0.5690 - val_recall0.7: 0.7656 - val_tp0.9
  : 2847371.0000 - val_fp0.9: 2085317.0000 - val_tn0.9: 19964284.0000 - val_fn0.9: 1317433.0000 - val_precision0.9: 0.5772 - val_recall0.9: 0.6837 - val_accuracy: 0.8719 - val_auc: 0.9139
  Epoch 42/50
  1280/1280 [==============================] - 1987s 2s/step - loss: 0.1985 - tp0.1: 8084262.2818 - fp0.1: 2648823.0039 - tn0.1: 41506455.0390 - fn0.1: 271123.8985 - precision0.1: 0.7515 - recall0.1: 0.9676 - tp0.3: 7842478.2342 - fp0.3:
   1315040.6995 - tn0.3: 42840230.2826 - fn0.3: 512907.9461 - precision0.3: 0.8555 - recall0.3: 0.9387 - tp0.5: 7555647.0101 - fp0.5: 708480.3888 - tn0.5: 43446801.7198 - fn0.5: 799739.1702 - precision0.5: 0.9137 - recall0.5: 0.9044 - tp
  0.7: 7164901.3419 - fp0.7: 337266.9789 - tn0.7: 43818009.3770 - fn0.7: 1190484.8384 - precision0.7: 0.9549 - recall0.7: 0.8575 - tp0.9: 6464041.4738 - fp0.9: 94869.2607 - tn0.9: 44060414.4278 - fn0.9: 1891344.7065 - precision0.9: 0.985
  5 - recall0.9: 0.7739 - accuracy: 0.9716 - auc: 0.9811 - val_loss: 0.5958 - val_tp0.1: 3933554.0000 - val_fp0.1: 2955615.0000 - val_tn0.1: 18990420.0000 - val_fn0.1: 334809.0000 - val_precision0.1: 0.5710 - val_recall0.1: 0.9216 - val_
  tp0.3: 3653243.0000 - val_fp0.3: 2331195.0000 - val_tn0.3: 19614844.0000 - val_fn0.3: 615120.0000 - val_precision0.3: 0.6105 - val_recall0.3: 0.8559 - val_tp0.5: 3388864.0000 - val_fp0.5: 1939453.0000 - val_tn0.5: 20006580.0000 - val_f
  n0.5: 879499.0000 - val_precision0.5: 0.6360 - val_recall0.5: 0.7939 - val_tp0.7: 3079796.0000 - val_fp0.7: 1608320.0000 - val_tn0.7: 20337718.0000 - val_fn0.7: 1188567.0000 - val_precision0.7: 0.6569 - val_recall0.7: 0.7215 - val_tp0.
  9: 2715008.0000 - val_fp0.9: 1171919.0000 - val_tn0.9: 20774120.0000 - val_fn0.9: 1553355.0000 - val_precision0.9: 0.6985 - val_recall0.9: 0.6361 - val_accuracy: 0.8925 - val_auc: 0.9240
  Epoch 43/50
  1280/1280 [==============================] - 1955s 2s/step - loss: 0.1987 - tp0.1: 8221268.6706 - fp0.1: 2670843.3326 - tn0.1: 41354847.4762 - fn0.1: 263685.1889 - precision0.1: 0.7553 - recall0.1: 0.9689 - tp0.3: 7974364.6284 - fp0.3:
   1305978.7268 - tn0.3: 42719745.8759 - fn0.3: 510589.2311 - precision0.3: 0.8607 - recall0.3: 0.9397 - tp0.5: 7690894.8962 - fp0.5: 703492.0546 - tn0.5: 43322229.0648 - fn0.5: 794058.9633 - precision0.5: 0.9176 - recall0.5: 0.9062 - tp
  0.7: 7300427.2842 - fp0.7: 340590.7377 - tn0.7: 43685101.9196 - fn0.7: 1184526.5753 - precision0.7: 0.9563 - recall0.7: 0.8599 - tp0.9: 6589497.3521 - fp0.9: 94983.5316 - tn0.9: 43930710.3302 - fn0.9: 1895456.5074 - precision0.9: 0.986
  2 - recall0.9: 0.7761 - accuracy: 0.9717 - auc: 0.9819 - val_loss: 0.6235 - val_tp0.1: 3909308.0000 - val_fp0.1: 3193225.0000 - val_tn0.1: 18785172.0000 - val_fn0.1: 326693.0000 - val_precision0.1: 0.5504 - val_recall0.1: 0.9229 - val_
  tp0.3: 3626074.0000 - val_fp0.3: 2618929.0000 - val_tn0.3: 19359480.0000 - val_fn0.3: 609927.0000 - val_precision0.3: 0.5806 - val_recall0.3: 0.8560 - val_tp0.5: 3338230.0000 - val_fp0.5: 2236986.0000 - val_tn0.5: 19741408.0000 - val_f
  n0.5: 897771.0000 - val_precision0.5: 0.5988 - val_recall0.5: 0.7881 - val_tp0.7: 2996830.0000 - val_fp0.7: 1846715.0000 - val_tn0.7: 20131684.0000 - val_fn0.7: 1239171.0000 - val_precision0.7: 0.6187 - val_recall0.7: 0.7075 - val_tp0.
  9: 2591293.0000 - val_fp0.9: 1332148.0000 - val_tn0.9: 20646248.0000 - val_fn0.9: 1644708.0000 - val_precision0.9: 0.6605 - val_recall0.9: 0.6117 - val_accuracy: 0.8804 - val_auc: 0.9182
  Epoch 44/50
  1280/1280 [==============================] - 1965s 2s/step - loss: 0.1994 - tp0.1: 8332024.2139 - fp0.1: 2677627.7361 - tn0.1: 41232032.9399 - fn0.1: 268956.1819 - precision0.1: 0.7573 - recall0.1: 0.9690 - tp0.3: 8088072.3755 - fp0.3:
   1334673.8392 - tn0.3: 42575000.3903 - fn0.3: 512908.0203 - precision0.3: 0.8587 - recall0.3: 0.9411 - tp0.5: 7804018.2927 - fp0.5: 727991.3505 - tn0.5: 43181681.2311 - fn0.5: 796962.1030 - precision0.5: 0.9148 - recall0.5: 0.9085 - tp
  0.7: 7411774.5628 - fp0.7: 351151.7166 - tn0.7: 43558520.9914 - fn0.7: 1189205.8329 - precision0.7: 0.9548 - recall0.7: 0.8632 - tp0.9: 6689648.7783 - fp0.9: 96689.1569 - tn0.9: 43812989.9953 - fn0.9: 1911331.6175 - precision0.9: 0.985
  8 - recall0.9: 0.7794 - accuracy: 0.9712 - auc: 0.9819 - val_loss: 0.5577 - val_tp0.1: 3767211.0000 - val_fp0.1: 2896593.0000 - val_tn0.1: 19207012.0000 - val_fn0.1: 343594.0000 - val_precision0.1: 0.5653 - val_recall0.1: 0.9164 - val_
  tp0.3: 3483824.0000 - val_fp0.3: 2230299.0000 - val_tn0.3: 19873292.0000 - val_fn0.3: 626981.0000 - val_precision0.3: 0.6097 - val_recall0.3: 0.8475 - val_tp0.5: 3200515.0000 - val_fp0.5: 1719966.0000 - val_tn0.5: 20383632.0000 - val_f
  n0.5: 910290.0000 - val_precision0.5: 0.6504 - val_recall0.5: 0.7786 - val_tp0.7: 2881571.0000 - val_fp0.7: 1322038.0000 - val_tn0.7: 20781552.0000 - val_fn0.7: 1229234.0000 - val_precision0.7: 0.6855 - val_recall0.7: 0.7010 - val_tp0.
  9: 2488169.0000 - val_fp0.9: 864656.0000 - val_tn0.9: 21238940.0000 - val_fn0.9: 1622636.0000 - val_precision0.9: 0.7421 - val_recall0.9: 0.6053 - val_accuracy: 0.8997 - val_auc: 0.9249
  Epoch 45/50
  1280/1280 [==============================] - 1967s 2s/step - loss: 0.2054 - tp0.1: 8039884.9914 - fp0.1: 2657730.7822 - tn0.1: 41537257.0453 - fn0.1: 275776.4333 - precision0.1: 0.7479 - recall0.1: 0.9656 - tp0.3: 7799006.9430 - fp0.3:
   1331880.2607 - tn0.3: 42863099.0406 - fn0.3: 516654.4817 - precision0.3: 0.8527 - recall0.3: 0.9360 - tp0.5: 7508282.6300 - fp0.5: 716584.7447 - tn0.5: 43478405.7619 - fn0.5: 807378.7947 - precision0.5: 0.9120 - recall0.5: 0.9006 - tp
  0.7: 7111748.2311 - fp0.7: 345919.2553 - tn0.7: 43849114.1967 - fn0.7: 1203913.1936 - precision0.7: 0.9530 - recall0.7: 0.8525 - tp0.9: 6406401.2319 - fp0.9: 97963.5379 - tn0.9: 44097016.4520 - fn0.9: 1909260.1928 - precision0.9: 0.984
  7 - recall0.9: 0.7671 - accuracy: 0.9708 - auc: 0.9799 - val_loss: 0.5755 - val_tp0.1: 3947505.0000 - val_fp0.1: 3387487.0000 - val_tn0.1: 18700156.0000 - val_fn0.1: 179254.0000 - val_precision0.1: 0.5382 - val_recall0.1: 0.9566 - val_
  tp0.3: 3738020.0000 - val_fp0.3: 2730917.0000 - val_tn0.3: 19356726.0000 - val_fn0.3: 388739.0000 - val_precision0.3: 0.5778 - val_recall0.3: 0.9058 - val_tp0.5: 3505678.0000 - val_fp0.5: 2359050.0000 - val_tn0.5: 19728584.0000 - val_f
  n0.5: 621081.0000 - val_precision0.5: 0.5978 - val_recall0.5: 0.8495 - val_tp0.7: 3196056.0000 - val_fp0.7: 1989619.0000 - val_tn0.7: 20098016.0000 - val_fn0.7: 930703.0000 - val_precision0.7: 0.6163 - val_recall0.7: 0.7745 - val_tp0.9
  : 2807022.0000 - val_fp0.9: 1495603.0000 - val_tn0.9: 20592028.0000 - val_fn0.9: 1319737.0000 - val_precision0.9: 0.6524 - val_recall0.9: 0.6802 - val_accuracy: 0.8863 - val_auc: 0.9356
  Epoch 46/50
  1280/1280 [==============================] - 1954s 2s/step - loss: 0.2006 - tp0.1: 8213399.9953 - fp0.1: 2647494.8681 - tn0.1: 41382018.9024 - fn0.1: 267742.8033 - precision0.1: 0.7532 - recall0.1: 0.9684 - tp0.3: 7974876.2194 - fp0.3:
   1321606.3544 - tn0.3: 42707910.7440 - fn0.3: 506266.5792 - precision0.3: 0.8563 - recall0.3: 0.9401 - tp0.5: 7686530.4450 - fp0.5: 705127.5909 - tn0.5: 43324387.6230 - fn0.5: 794612.3536 - precision0.5: 0.9155 - recall0.5: 0.9060 - tp
  0.7: 7293262.6581 - fp0.7: 337606.6417 - tn0.7: 43691898.0304 - fn0.7: 1187880.1405 - precision0.7: 0.9556 - recall0.7: 0.8595 - tp0.9: 6593016.1983 - fp0.9: 93813.7900 - tn0.9: 43935696.6682 - fn0.9: 1888126.6003 - precision0.9: 0.985
  9 - recall0.9: 0.7775 - accuracy: 0.9713 - auc: 0.9815 - val_loss: 0.5779 - val_tp0.1: 3847918.0000 - val_fp0.1: 3630042.0000 - val_tn0.1: 18523056.0000 - val_fn0.1: 213382.0000 - val_precision0.1: 0.5146 - val_recall0.1: 0.9475 - val_
  tp0.3: 3624637.0000 - val_fp0.3: 2927247.0000 - val_tn0.3: 19225848.0000 - val_fn0.3: 436663.0000 - val_precision0.3: 0.5532 - val_recall0.3: 0.8925 - val_tp0.5: 3387759.0000 - val_fp0.5: 2442181.0000 - val_tn0.5: 19710918.0000 - val_f
  n0.5: 673541.0000 - val_precision0.5: 0.5811 - val_recall0.5: 0.8342 - val_tp0.7: 3093355.0000 - val_fp0.7: 1959305.0000 - val_tn0.7: 20193798.0000 - val_fn0.7: 967945.0000 - val_precision0.7: 0.6122 - val_recall0.7: 0.7617 - val_tp0.9
  : 2709194.0000 - val_fp0.9: 1315902.0000 - val_tn0.9: 20837192.0000 - val_fn0.9: 1352106.0000 - val_precision0.9: 0.6731 - val_recall0.9: 0.6671 - val_accuracy: 0.8811 - val_auc: 0.9327
  Epoch 47/50
  1280/1280 [==============================] - 1945s 2s/step - loss: 0.1969 - tp0.1: 8300923.1351 - fp0.1: 2610298.6073 - tn0.1: 41331766.4083 - fn0.1: 267669.1296 - precision0.1: 0.7642 - recall0.1: 0.9694 - tp0.3: 8061644.4340 - fp0.3:
   1313054.2631 - tn0.3: 42629006.5870 - fn0.3: 506947.8306 - precision0.3: 0.8611 - recall0.3: 0.9422 - tp0.5: 7779027.9711 - fp0.5: 711910.2787 - tn0.5: 43230141.7112 - fn0.5: 789564.2935 - precision0.5: 0.9166 - recall0.5: 0.9097 - tp
  0.7: 7390637.0796 - fp0.7: 345621.5621 - tn0.7: 43596452.1616 - fn0.7: 1177955.1850 - precision0.7: 0.9553 - recall0.7: 0.8650 - tp0.9: 6688557.9680 - fp0.9: 97298.4106 - tn0.9: 43844794.7112 - fn0.9: 1880034.2966 - precision0.9: 0.985
  5 - recall0.9: 0.7835 - accuracy: 0.9714 - auc: 0.9821 - val_loss: 0.6259 - val_tp0.1: 3930823.0000 - val_fp0.1: 3136289.0000 - val_tn0.1: 18868696.0000 - val_fn0.1: 278584.0000 - val_precision0.1: 0.5562 - val_recall0.1: 0.9338 - val_
  tp0.3: 3682497.0000 - val_fp0.3: 2597221.0000 - val_tn0.3: 19407772.0000 - val_fn0.3: 526910.0000 - val_precision0.3: 0.5864 - val_recall0.3: 0.8748 - val_tp0.5: 3429169.0000 - val_fp0.5: 2237642.0000 - val_tn0.5: 19767344.0000 - val_f
  n0.5: 780238.0000 - val_precision0.5: 0.6051 - val_recall0.5: 0.8146 - val_tp0.7: 3114765.0000 - val_fp0.7: 1889104.0000 - val_tn0.7: 20115888.0000 - val_fn0.7: 1094642.0000 - val_precision0.7: 0.6225 - val_recall0.7: 0.7400 - val_tp0.
  9: 2710793.0000 - val_fp0.9: 1416123.0000 - val_tn0.9: 20588868.0000 - val_fn0.9: 1498614.0000 - val_precision0.9: 0.6569 - val_recall0.9: 0.6440 - val_accuracy: 0.8849 - val_auc: 0.9224
  Epoch 48/50
  1280/1280 [==============================] - 1953s 2s/step - loss: 0.1928 - tp0.1: 8105003.8283 - fp0.1: 2580803.4801 - tn0.1: 41568825.8158 - fn0.1: 256048.8673 - precision0.1: 0.7573 - recall0.1: 0.9697 - tp0.3: 7868496.5355 - fp0.3:
   1297144.6347 - tn0.3: 42852437.2209 - fn0.3: 492556.1600 - precision0.3: 0.8575 - recall0.3: 0.9414 - tp0.5: 7589914.3154 - fp0.5: 704118.4832 - tn0.5: 43445489.9368 - fn0.5: 771138.3802 - precision0.5: 0.9146 - recall0.5: 0.9080 - tp
  0.7: 7204444.4348 - fp0.7: 338098.1210 - tn0.7: 43811497.8689 - fn0.7: 1156608.2607 - precision0.7: 0.9549 - recall0.7: 0.8617 - tp0.9: 6516177.3638 - fp0.9: 93695.1686 - tn0.9: 44055912.6136 - fn0.9: 1844875.3318 - precision0.9: 0.985
  8 - recall0.9: 0.7794 - accuracy: 0.9719 - auc: 0.9822 - val_loss: 0.5549 - val_tp0.1: 3956721.0000 - val_fp0.1: 3232246.0000 - val_tn0.1: 18771428.0000 - val_fn0.1: 254013.0000 - val_precision0.1: 0.5504 - val_recall0.1: 0.9397 - val_
  tp0.3: 3708830.0000 - val_fp0.3: 2603884.0000 - val_tn0.3: 19399780.0000 - val_fn0.3: 501904.0000 - val_precision0.3: 0.5875 - val_recall0.3: 0.8808 - val_tp0.5: 3455265.0000 - val_fp0.5: 2129156.0000 - val_tn0.5: 19874504.0000 - val_f
  n0.5: 755469.0000 - val_precision0.5: 0.6187 - val_recall0.5: 0.8206 - val_tp0.7: 3150570.0000 - val_fp0.7: 1691047.0000 - val_tn0.7: 20312616.0000 - val_fn0.7: 1060164.0000 - val_precision0.7: 0.6507 - val_recall0.7: 0.7482 - val_tp0.
  9: 2734733.0000 - val_fp0.9: 1128005.0000 - val_tn0.9: 20875662.0000 - val_fn0.9: 1476001.0000 - val_precision0.9: 0.7080 - val_recall0.9: 0.6495 - val_accuracy: 0.8900 - val_auc: 0.9326
  Epoch 49/50
  1280/1280 [==============================] - 1952s 2s/step - loss: 0.1904 - tp0.1: 8205918.2201 - fp0.1: 2597375.0523 - tn0.1: 41463883.8283 - fn0.1: 243494.6815 - precision0.1: 0.7596 - recall0.1: 0.9713 - tp0.3: 7967288.4333 - fp0.3:
   1294707.1905 - tn0.3: 42766530.0679 - fn0.3: 482124.4684 - precision0.3: 0.8602 - recall0.3: 0.9426 - tp0.5: 7690726.4028 - fp0.5: 706852.2178 - tn0.5: 43354411.7049 - fn0.5: 758686.4988 - precision0.5: 0.9160 - recall0.5: 0.9099 - tp
  0.7: 7310265.9742 - fp0.7: 341418.8126 - tn0.7: 43719824.0367 - fn0.7: 1139146.9274 - precision0.7: 0.9557 - recall0.7: 0.8651 - tp0.9: 6619945.5667 - fp0.9: 95254.3942 - tn0.9: 43965990.5909 - fn0.9: 1829467.3349 - precision0.9: 0.986
  3 - recall0.9: 0.7836 - accuracy: 0.9719 - auc: 0.9831 - val_loss: 0.7113 - val_tp0.1: 3891189.0000 - val_fp0.1: 3192765.0000 - val_tn0.1: 18774752.0000 - val_fn0.1: 355695.0000 - val_precision0.1: 0.5493 - val_recall0.1: 0.9162 - val_
  tp0.3: 3627202.0000 - val_fp0.3: 2690459.0000 - val_tn0.3: 19277064.0000 - val_fn0.3: 619682.0000 - val_precision0.3: 0.5741 - val_recall0.3: 0.8541 - val_tp0.5: 3369672.0000 - val_fp0.5: 2384023.0000 - val_tn0.5: 19583492.0000 - val_f
  n0.5: 877212.0000 - val_precision0.5: 0.5857 - val_recall0.5: 0.7934 - val_tp0.7: 3076485.0000 - val_fp0.7: 2107024.0000 - val_tn0.7: 19860492.0000 - val_fn0.7: 1170399.0000 - val_precision0.7: 0.5935 - val_recall0.7: 0.7244 - val_tp0.
  9: 2710033.0000 - val_fp0.9: 1728311.0000 - val_tn0.9: 20239206.0000 - val_fn0.9: 1536851.0000 - val_precision0.9: 0.6106 - val_recall0.9: 0.6381 - val_accuracy: 0.8756 - val_auc: 0.9085
  Epoch 50/50
  1280/1280 [==============================] - 1959s 2s/step - loss: 0.1904 - tp0.1: 8168142.8899 - fp0.1: 2531078.8610 - tn0.1: 41557584.1272 - fn0.1: 253833.5800 - precision0.1: 0.7637 - recall0.1: 0.9699 - tp0.3: 7935448.9009 - fp0.3:
   1268230.0484 - tn0.3: 42820446.9766 - fn0.3: 486527.5691 - precision0.3: 0.8629 - recall0.3: 0.9420 - tp0.5: 7667088.7853 - fp0.5: 699134.4457 - tn0.5: 43389534.5613 - fn0.5: 754887.6846 - precision0.5: 0.9167 - recall0.5: 0.9102 - tp
  0.7: 7290487.3372 - fp0.7: 342110.9336 - tn0.7: 43746573.7869 - fn0.7: 1131489.1327 - precision0.7: 0.9553 - recall0.7: 0.8655 - tp0.9: 6585906.2607 - fp0.9: 97053.1694 - tn0.9: 43991631.9688 - fn0.9: 1836070.2092 - precision0.9: 0.985
  5 - recall0.9: 0.7818 - accuracy: 0.9724 - auc: 0.9826 - val_loss: 0.5552 - val_tp0.1: 3972788.0000 - val_fp0.1: 3022431.0000 - val_tn0.1: 18950448.0000 - val_fn0.1: 268737.0000 - val_precision0.1: 0.5679 - val_recall0.1: 0.9366 - val_
  tp0.3: 3744205.0000 - val_fp0.3: 2467134.0000 - val_tn0.3: 19505742.0000 - val_fn0.3: 497320.0000 - val_precision0.3: 0.6028 - val_recall0.3: 0.8827 - val_tp0.5: 3515241.0000 - val_fp0.5: 2098091.0000 - val_tn0.5: 19874784.0000 - val_f
  n0.5: 726284.0000 - val_precision0.5: 0.6262 - val_recall0.5: 0.8288 - val_tp0.7: 3227939.0000 - val_fp0.7: 1707258.0000 - val_tn0.7: 20265616.0000 - val_fn0.7: 1013586.0000 - val_precision0.7: 0.6541 - val_recall0.7: 0.7610 - val_tp0.
  9: 2849341.0000 - val_fp0.9: 1228067.0000 - val_tn0.9: 20744808.0000 - val_fn0.9: 1392184.0000 - val_precision0.9: 0.6988 - val_recall0.9: 0.6718 - val_accuracy: 0.8923 - val_auc: 0.9339
  420/420 [==============================] - 126s 300ms/step - loss: 0.5623 - tp0.1: 4784751.0000 - fp0.1: 3873965.0000 - tn0.1: 25422992.0000 - fn0.1: 324680.0000 - precision0.1: 0.5526 - recall0.1: 0.9365 - tp0.3: 4514069.0000 - fp0.3:
   3235942.0000 - tn0.3: 26061030.0000 - fn0.3: 595362.0000 - precision0.3: 0.5825 - recall0.3: 0.8835 - tp0.5: 4242444.0000 - fp0.5: 2843465.0000 - tn0.5: 26453508.0000 - fn0.5: 866987.0000 - precision0.5: 0.5987 - recall0.5: 0.8303 - t
  p0.7: 3892738.0000 - fp0.7: 2409415.0000 - tn0.7: 26887552.0000 - fn0.7: 1216693.0000 - precision0.7: 0.6177 - recall0.7: 0.7619 - tp0.9: 3413153.0000 - fp0.9: 1853895.0000 - tn0.9: 27443078.0000 - fn0.9: 1696278.0000 - precision0.9: 0
  .6480 - recall0.9: 0.6680 - accuracy: 0.8922 - auc: 0.9306
  2021/02/06 23:57:14 INFO mlflow.projects: === Run (ID 'b9935d1e554c423fb2852242f4c4504c') succeeded ===
  (tf-nightly) [ye53nis@node221 drmed-git]$
  

• test data is not saved out automatically, but can be copied from the log above

  420/420 [==============================] - 126s 300ms/step - loss: 0.5623

  …	0.1	0.3	0.5	0.7	0.9
  tp	4784751.0000	4514069.0000	4242444.0000	3892738.0000	3413153.0000
  fp	3873965.0000	3235942.0000	2843465.0000	2409415.0000	1853895.0000
  fn	324680.0000	595362.0000	866987.0000	1216693.0000	1696278.0000
  tn	25422992.0000	26061030.0000	26453508.0000	26887552.0000	27443078.0000
  all	34,406,388	34,406,403	34,406,404	34,406,398	34,406,404
  precision	0.5526	0.5825	0.5987	0.6177	0.6480
  recall	0.9365	0.8835	0.8303	0.7619	0.6680

  accuracy: 0.8922 - auc: 0.9306

  0.1	actual positive	actual negative
  pred positive	0.13906577	0.11259435	Prec: 0.5526
  pred negative	9.4345271e-3	0.73890325
  	Recall: 0.9365
  			F1: 0.69506400

  0.3	actual positive	actual negative
  pred positive	0.13119852	0.094050575	Prec: 0.5825
  pred negative	0.017303814	0.75744709
  	Recall: 0.8835
  			F1: 0.70209925

  0.5	actual positive	actual negative
  pred positive	0.12330391	0.082643481	Prec: 0.5987
  pred negative	0.025198419	0.76885419
  	Recall: 0.8303
  			F1: 0.69573213

  0.7	actual positive	actual negative
  pred positive	0.11313995	0.070028109	Prec: 0.6177
  pred negative	0.035362406	0.78146954
  	Recall: 0.7619
  			F1: 0.68226389

  0.9	actual positive	actual negative
  pred positive	0.099201096	0.053882266	Prec: 0.6480
  pred negative	0.049301229	0.79761541
  	Recall: 0.6680
  			F1: 0.65784802

after epoch 1: epoch 2: epoch 3: epoch 4: epoch 5: epoch 10: epoch 20: epoch 30: epoch 40: epoch 50:

1. Load the required modules

   %cd /beegfs/ye53nis/drmed-git
   

   /beegfs/ye53nis/drmed-git
   

   from pathlib import Path
   
   import sys
   import mlflow
   import matplotlib.pyplot as plt
   import numpy as np
   import pandas as pd
   import seaborn as sns
   mlflow.version.VERSION
   

   1.13.1
   

   sys.path.append('src/')
   from fluotracify.simulations import (
      import_simulation_from_csv as isfc,
      analyze_simulations as ans,
   )
   from fluotracify.training import build_model as bm, preprocess_data as ppd
   from fluotracify.applications import correlate, plots, correction
   from fluotracify.imports import ptu_utils as ptu
   

   import importlib
   importlib.reload(correction)
   

   <module 'fluotracify.applications.correction' from 'src/fluotracify/applications/correction.py'>
   

folder = '/beegfs/ye53nis/saves/firstartifact_Nov2020_test_run1/'
col_per_example = 3
lab_thresh = 0.04
pred_thresh = 0.5
xunit = 1
artifact = 0
model_type = 1
fwhm = 250
run_id = 'b9935d1e554c423fb2852242f4c4504c'
length_delimiter = 2**14

• now load the trained model

  mlflow.set_tracking_uri('file:///beegfs/ye53nis/drmed-git/data/mlruns')
  client = mlflow.tracking.MlflowClient(tracking_uri=mlflow.get_tracking_uri())
  model_path = client.download_artifacts(run_id=run_id,
                                         path='model')
  model_keras = mlflow.keras.load_model(model_uri=model_path,
                                        custom_objects={'binary_ce_dice':bm.binary_ce_dice_loss()})
  print(model_path, '\n', model_keras)
  

  /beegfs/ye53nis/drmed-git/data/mlruns/3/b9935d1e554c423fb2852242f4c4504c/artifacts/model
   <tensorflow.python.keras.engine.functional.Functional object at 0x2ae3e21c2d60>
  

• I copied the test data which was randomly sampled in the mlflow run (See docs above) in an extra directory. Next time, train and test data will be split beforehand.

  dataset, _, nsamples, experiment_params = isfc.import_from_csv(
      folder=folder,
      header=12,
      frac_train=1,
      col_per_example=col_per_example,
      dropindex=None,
      dropcolumns=None)
  experiment_params
  

  train 0 /beegfs/ye53nis/saves/firstartifact_Nov2020_test_run1/traces_brightclust_Nov2020_D3.0_set009.csv
  train 1 /beegfs/ye53nis/saves/firstartifact_Nov2020_test_run1/traces_brightclust_Nov2020_D0.08_set003.csv
  train 2 /beegfs/ye53nis/saves/firstartifact_Nov2020_test_run1/traces_brightclust_Nov2020_D50_set003.csv
  train 3 /beegfs/ye53nis/saves/firstartifact_Nov2020_test_run1/traces_brightclust_Nov2020_D0.4_set007.csv
  train 4 /beegfs/ye53nis/saves/firstartifact_Nov2020_test_run1/traces_brightclust_Nov2020_D0.6_set009.csv
  train 5 /beegfs/ye53nis/saves/firstartifact_Nov2020_test_run1/traces_brightclust_Nov2020_D3.0_set008.csv
  train 6 /beegfs/ye53nis/saves/firstartifact_Nov2020_test_run1/traces_brightclust_Nov2020_D0.2_set010.csv
  train 7 /beegfs/ye53nis/saves/firstartifact_Nov2020_test_run1/traces_brightclust_Nov2020_D1.0_set008.csv
  train 8 /beegfs/ye53nis/saves/firstartifact_Nov2020_test_run1/traces_brightclust_Nov2020_D0.2_set007.csv
  train 9 /beegfs/ye53nis/saves/firstartifact_Nov2020_test_run1/traces_brightclust_Nov2020_D0.1_set002.csvtrain 10 /beegfs/ye53nis/saves/firstartifact_Nov2020_test_run1/traces_brightclust_Nov2020_D0.2_set008.csv
  train 11 /beegfs/ye53nis/saves/firstartifact_Nov2020_test_run1/traces_brightclust_Nov2020_D10_set001.csv
  train 12 /beegfs/ye53nis/saves/firstartifact_Nov2020_test_run1/traces_brightclust_Nov2020_D0.4_set008.csv
  train 13 /beegfs/ye53nis/saves/firstartifact_Nov2020_test_run1/traces_brightclust_Nov2020_D0.4_set005.csv
  train 14 /beegfs/ye53nis/saves/firstartifact_Nov2020_test_run1/traces_brightclust_Nov2020_D3.0_set004.csv
  train 15 /beegfs/ye53nis/saves/firstartifact_Nov2020_test_run1/traces_brightclust_Nov2020_D1.0_set005.csv
  train 16 /beegfs/ye53nis/saves/firstartifact_Nov2020_test_run1/traces_brightclust_Nov2020_D0.1_set006.csv
  train 17 /beegfs/ye53nis/saves/firstartifact_Nov2020_test_run1/traces_brightclust_Nov2020_D0.069_set010.csv
  train 18 /beegfs/ye53nis/saves/firstartifact_Nov2020_test_run1/traces_brightclust_Nov2020_D3.0_set002.csv
  train 19 /beegfs/ye53nis/saves/firstartifact_Nov2020_test_run1/traces_brightclust_Nov2020_D1.0_set006.csv
  

  	0	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19
  unique identifier	340ad302-cf12-4224-addf-9455009bc9b3	b85aeec3-5e79-444f-b0c5-5ba7057fad8c	b2b78a0f-278b-4b7d-82a3-5e625fbc91de	32d6e80c-92af-48c2-a756-1eee94d17c74	87d9374a-3ae4-4dfb-8dc3-ec401df4961d	90ec2bbf-d308-4895-b5fd-1974c0b3390d	5c310958-a0cc-4d1d-b721-f8652afd9707	b5d1537c-4113-4f13-a1ca-65664181f8cc	33ee5002-7064-4882-88de-79c773d741cd	9a2f89bd-c64e-4324-910c-46a5b8a657ba	71894018-b950-4672-a182-653d59f3b7db	6a3c00e3-24e6-4758-9c6d-f2a9dc23f7d9	20f0a9df-2c4d-409f-86de-ceda40d1aafe	eb3f3035-955f-4e86-bfbd-16421eede63f	cf57c545-933f-41b8-bcfe-19723d79b7bf	84116b56-7264-4f63-833a-2fa1af968288	55e826c0-a09a-4f1e-8f57-ddc113715aaf	bebf4d6b-0ffe-4dcf-be78-93bfa70dd04c	f2dcf7ee-f180-4b26-9706-526ceb6b9fe6	c1a85eb4-f202-40f7-8bce-cd3fd3a843dc
  path and file name	/beegfs/ye53nis/saves/firstartifact\Nov2020/3….	/beegfs/ye53nis/saves/firstartifact\Nov2020/0….	/beegfs/ye53nis/saves/firstartifact\Nov2020/50…	/beegfs/ye53nis/saves/firstartifact\Nov2020/0….	/beegfs/ye53nis/saves/firstartifact\Nov2020/0….	/beegfs/ye53nis/saves/firstartifact\Nov2020/3….	/beegfs/ye53nis/saves/firstartifact\Nov2020/0….	/beegfs/ye53nis/saves/firstartifact\Nov2020/1….	/beegfs/ye53nis/saves/firstartifact\Nov2020/0….	/beegfs/ye53nis/saves/firstartifact\Nov2020/0….	/beegfs/ye53nis/saves/firstartifact\Nov2020/0….	/beegfs/ye53nis/saves/firstartifact\Nov2020/10…	/beegfs/ye53nis/saves/firstartifact\Nov2020/0….	/beegfs/ye53nis/saves/firstartifact\Nov2020/0….	/beegfs/ye53nis/saves/firstartifact\Nov2020/3….	/beegfs/ye53nis/saves/firstartifact\Nov2020/1….	/beegfs/ye53nis/saves/firstartifact\Nov2020/0….	/beegfs/ye53nis/saves/firstartifact\Nov2020/0….	/beegfs/ye53nis/saves/firstartifact\Nov2020/3….	/beegfs/ye53nis/saves/firstartifact\Nov2020/1….
  FWHMs of excitation PSFs used in nm	[250]	[250]	[250]	[250]	[250]	[250]	[250]	[250]	[250]	[250]	[250]	[250]	[250]	[250]	[250]	[250]	[250]	[250]	[250]	[250]
  Extent of simulated PSF (distance to center of Gaussian) in nm	4000	4000	4000	4000	4000	4000	4000	4000	4000	4000	4000	4000	4000	4000	4000	4000	4000	4000	4000	4000
  total simulation time in ms	16384	16384	16384	16384	16384	16384	16384	16384	16384	16384	16384	16384	16384	16384	16384	16384	16384	16384	16384	16384
  time step in ms	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0
  number of fast molecules	2422	856	2561	2887	1296	766	1247	3300	895	2975	1239	1376	1724	560	1785	2059	2060	679	2900	1357
  diffusion rate of molecules in micrometer\2 / s	3.0	0.08	50	0.4	0.6	3.0	0.2	1.0	0.2	0.1	0.2	10	0.4	0.4	3.0	1.0	0.1	0.069	3.0	1.0
  width of the simulation in nm	3000.0	3000.0	3000.0	3000.0	3000.0	3000.0	3000.0	3000.0	3000.0	3000.0	3000.0	3000.0	3000.0	3000.0	3000.0	3000.0	3000.0	3000.0	3000.0	3000.0
  height of the simulation in nm	3000.0	3000.0	3000.0	3000.0	3000.0	3000.0	3000.0	3000.0	3000.0	3000.0	3000.0	3000.0	3000.0	3000.0	3000.0	3000.0	3000.0	3000.0	3000.0	3000.0
  number of slow clusters	3	7	7	3	3	10	3	3	7	10	7	7	10	3	10	3	10	3	3	10
  diffusion rate of clusters in micrometer\2 / s	1.0	0.1	0.1	1.0	1.0	0.01	1.0	1.0	0.1	0.01	0.1	0.1	0.01	1.0	0.01	1.0	0.01	1.0	1.0	0.01
  trace001	label001\1	label001\1	label001\1	label001\1	label001\1	label001\1	label001\1	label001\1	label001\1	label001\1	label001\1	label001\1	label001\1	label001\1	label001\1	label001\1	label001\1	label001\1	label001\1	label001\1

  diffrates = experiment_params.loc['diffusion rate of molecules in micrometer^2 / s'].astype(np.float32)
  nmols = experiment_params.loc['number of fast molecules'].astype(np.float32)
  clusters = experiment_params.loc['diffusion rate of clusters in micrometer^2 / s'].astype(np.float32)
  
  dataset_sep = isfc.separate_data_and_labels(array=dataset,
                                              nsamples=nsamples,
                                              col_per_example=col_per_example)
  
  features = dataset_sep['0']
  labels_artifact = dataset_sep['1']
  labels_artifact_bool = labels_artifact > lab_thresh
  labels_puretrace = dataset_sep['2']
  

  The given DataFrame was split into 3 parts with shapes: [(16384, 2000), (16384, 2000), (16384, 2000)]
  

• First attempt: look at distribution of all correlated diffrates by each simulated diffusion rate. The following function was thus executed for each simulated diffusion time and the box plots put together in inkscape by hand.

  pred_thresh = 0.5
  diffrate_of_interest = 50
  transit_time_expected = ((float(fwhm) / 1000)**2 * 1000) / (
      diffrate_of_interest * 8 * np.log(2.0))
  upper_bin = np.ceil(4*transit_time_expected)
  xunit_bins = np.arange(0, upper_bin, upper_bin / 50)
  
  out = plots.plot_distribution_of_correlations_corrected_by_prediction(
      diffrate_of_interest=diffrate_of_interest,
      model=model_keras,
      lab_thresh=lab_thresh,
      pred_thresh=pred_thresh,
      xunit=xunit,
      artifact=artifact,
      model_type=model_type,
      xunit_bins=xunit_bins,
      experiment_params=experiment_params,
      nsamples=nsamples,
      features=features,
      labels_artifact=labels_artifact,
      labels_puretrace=labels_puretrace,
      number_of_traces=100)
  

  

• We see some interesting results in the corrupted data without correction (first box, in blue), can they be explained by the test data?

  speed of clusters	0.01	0.1	1.0
  0.069 test run1			10
  0.08 test run1		3
  0.1 test run1	2, 6
  0.2 test run1		7, 8	10
  0.4 test run1	8		5, 7
  0.6 test run1			9
  1.0 test run1	6		5, 8
  3.0 test run1	4, 8		2, 9
  10 test run1		1
  50 test run1		3

  • subsets which look like the bright clusters lead to a shift in the correlation curves towards their own transit time:
    • 0.069=163.35ms: cluster speed 1.0=11.27ms, median correlated speed 11.6ms
    • 0.08=140.89ms: cluster speed 0.1=112.71ms, median correlated speed 100.5ms
    • 0.6=18.79ms: cluster speed 1.0=11.27ms, median correlated speed 12.10ms
    • 10=1.13ms: cluster speed 0.1=112.71ms, median correlated speed 92.1ms
    • 50=0.23ms: cluster speed 0.1=112.71ms, median correlated speed 96.5ms
  • subsets which fall out of the pattern where I have to take a closer look:
    • 0.1=112.71ms: cluster speed 0.01=1127ms, median correlated speed 346.6ms → possible explanation: it is a rule of thumb in the FCS community, that you need around 2-3 orders of magnitude of measurement time longer than transit times you want to measure. For speeds of 1.13s that means measurement times of 113-1130s, where we are far away from with our simulated ~16s of trace.
    • 0.2=56.36ms: cluster speed 0.1=112.71ms AND 1.0=11.27ms, median correlated speed 73ms → possible explanation: there seem to be two maxima in the distribution, one around 10ms, one around 70-110ms, which kind of fits to the two kinds of clusters in these traces. → another strange thing: it has a quite long tail of transit times, way over 200ms, which does not seem related to the direct cluster speeds.
    • 0.4=28.18ms, 1.0=11.27ms and 3.0=3.76ms: cluster speed 0.01=1127ms AND 1.0=11.27ms, median correlated speed 14.4ms → strange case, since the traces with the slower 0.01 clusters do not show up in the distribution! The log of isfc.import_from_csv shows, that the respective files were loaded and used for these plots.
• Looking at the predictions (and controls)
  • subsets which look fine, as in: predictions ~ controls
    • 0.069, 0.08, 0.2, 0.6
  • subsets where there seem to be a subsubset of traces where the prediction fails:
    • 0.1: median is fine, but the distribution is a lot wider, including a considerable left-ward skew, which does not show in either of the other distributions (corrupted or controls)
    • 0.4: median is fine, distribution is also okay, but a subset of traces gets very low transit times after correlation → have to figure out which subset this is
    • 1.0, 3.0: median is fine, but distribution has a considerable left-ward skew, and quite some outliers on the right
    • 10, 50: median and distribution are okay, BUT here there is quite a difference between the distribution in predictions, label control and pure control → here, most clearly in 50, it seems artifacts are introduced through the correction method.

corr_out = ans.correlate_simulations_corrected_by_prediction(
    model=model_keras,
    lab_thresh=lab_thresh,
    pred_thresh=pred_thresh,
    artifact=artifact,
    model_type=model_type,
    experiment_params=experiment_params,
    nsamples=nsamples,
    features=features,
    labels_artifact=labels_artifact,
    labels_puretrace=labels_puretrace,
    save_as_csv=True)
corr_out

processed correlation of 2000 traces with correction by label
processed correlation of 1999 traces with correction by prediction
processed correlation of 2000 traces without correction
processed correlation of pure 2000 traces

8000 rows × 7 columns

corr_out = pd.read_csv(filepath_or_buffer='data/exp-210204-unet/2021-02-20_correlations.csv')
corr_out

8000 rows × 7 columns

Awesome! Now let's try categorical plotting to examine the results in a more structured manner. For the following 3 plots, I just changed the x value.

ax = sns.catplot(data=corr_out,
      kind='strip',
      x='$D$ in $\\frac{{\mu m^2}}{{s}}$',
      y='Traces used',
      hue='Simulated $D_{{clust}}$',
      col='Simulated $D$',
      col_wrap=3,
      orient='h',
      dodge=True,
      sharex=False)
plt.show()

Trace lengths: Transit times: Diffusion rates:

I have the theory, that trace lengths have quite an impact on transit times and diffusion rates. Let's try to do a big plot of transittimes vs trace lengths and diffusion rates against trace lengths.

Here I used the code in a jupyter notebook to produce this plot:

1. There is a problem with the UNet, if it has to classify features different than the training size (1, 16384, 1) for the first time. It gives the following error:

   ---------------------------------------------------------------------------
   ValueError                                Traceback (most recent call last)
   <ipython-input-7-4c721240eb1b> in <module>
         3 length_delimiter = 2**13  # for U-Net
         4 bin_for_correlation = 1e5
   ----> 5 out = correction.correct_experimental_traces_from_ptu_by_unet_prediction(
         6   path_list=[path_tb_pex5_egfp],
         7   model=model_keras,
   
   /beegfs/ye53nis/drmed-git/src/fluotracify/applications/correction.py in correct_experimental_traces_from_ptu_by_unet_prediction(path_list, model, pred_thresh, photon_count_bin, ntraces, save_as_csv)
       405             print('Processing correlation with correction by prediction '
       406                   'of dataset {}'.format(i + 1))
   --> 407             data['{}-pred'.format(i)] = correct_correlation_by_unet_prediction(
       408                 ntraces=ntraces,
       409                 traces_of_interest=ptu_1ms.astype(np.float64),
   
   /beegfs/ye53nis/drmed-git/src/fluotracify/applications/correction.py in correct_correlation_by_unet_prediction(ntraces, traces_of_interest, model, pred_thresh, fwhm, length_delimiter, traces_for_correlation, bin_for_correlation, verbose)
       215             ntraces_index=ntraces_index)
       216
   --> 217         predictions = model.predict(features_prepro, verbose=0)
       218         predictions = predictions.flatten()
       219
   
   ~/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py in predict(self, x, batch_size, verbose, steps, callbacks, max_queue_size, workers, use_multiprocessing)
      1661           for step in data_handler.steps():
      1662             callbacks.on_predict_batch_begin(step)
   -> 1663             tmp_batch_outputs = self.predict_function(iterator)
      1664             if data_handler.should_sync:
      1665               context.async_wait()
   
   ~/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py in __call__(self, *args, **kwds)
       816     tracing_count = self.experimental_get_tracing_count()
       817     with trace.Trace(self._name) as tm:
   --> 818       result = self._call(*args, **kwds)
       819       compiler = "xla" if self._jit_compile else "nonXla"
       820       new_tracing_count = self.experimental_get_tracing_count()
   
   ~/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py in _call(self, *args, **kwds)
       860       # This is the first call of __call__, so we have to initialize.
       861       initializers = []
   --> 862       self._initialize(args, kwds, add_initializers_to=initializers)
       863     finally:
       864       # At this point we know that the initialization is complete (or less
   
   ~/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py in _initialize(self, args, kwds, add_initializers_to)
       701     self._graph_deleter = FunctionDeleter(self._lifted_initializer_graph)
       702     self._concrete_stateful_fn = (
   --> 703         self._stateful_fn._get_concrete_function_internal_garbage_collected(  # pylint: disable=protected-access
       704             *args, **kwds))
       705
   
   ~/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/eager/function.py in _get_concrete_function_internal_garbage_collected(self, *args, **kwargs)
      3018       args, kwargs = None, None
      3019     with self._lock:
   -> 3020       graph_function, _ = self._maybe_define_function(args, kwargs)
      3021     return graph_function
      3022
   
   ~/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/eager/function.py in _maybe_define_function(self, args, kwargs)
      3412
      3413           self._function_cache.missed.add(call_context_key)
   -> 3414           graph_function = self._create_graph_function(args, kwargs)
      3415           self._function_cache.primary[cache_key] = graph_function
      3416
   
   ~/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/eager/function.py in _create_graph_function(self, args, kwargs, override_flat_arg_shapes)
      3247     arg_names = base_arg_names + missing_arg_names
      3248     graph_function = ConcreteFunction(
   -> 3249         func_graph_module.func_graph_from_py_func(
      3250             self._name,
      3251             self._python_function,
   
   ~/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/framework/func_graph.py in func_graph_from_py_func(name, python_func, args, kwargs, signature, func_graph, autograph, autograph_options, add_control_dependencies, arg_names, op_return_value, collections, capture_by_value, override_flat_arg_shapes)
       996         _, original_func = tf_decorator.unwrap(python_func)
       997
   --> 998       func_outputs = python_func(*func_args, **func_kwargs)
       999
      1000       # invariant: `func_outputs` contains only Tensors, CompositeTensors,
   
   ~/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py in wrapped_fn(*args, **kwds)
       610             xla_context.Exit()
       611         else:
   --> 612           out = weak_wrapped_fn().__wrapped__(*args, **kwds)
       613         return out
       614
   
    ~/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/framework/func_graph.py in wrapper(*args, **kwargs)
       983           except Exception as e:  # pylint:disable=broad-except
       984             if hasattr(e, "ag_error_metadata"):
   --> 985               raise e.ag_error_metadata.to_exception(e)
       986             else:
       987               raise
   
   ValueError: in user code:
   
       /home/ye53nis/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py:1512 predict_function  *
           return step_function(self, iterator)
       /home/ye53nis/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py:1502 step_function  **
           outputs = model.distribute_strategy.run(run_step, args=(data,))
       /home/ye53nis/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/distribute/distribute_lib.py:1262 run
           return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
       /home/ye53nis/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/distribute/distribute_lib.py:2734 call_for_each_replica
           return self._call_for_each_replica(fn, args, kwargs)
       /home/ye53nis/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/distribute/distribute_lib.py:3423 _call_for_each_replica
           return fn(*args, **kwargs)
       /home/ye53nis/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py:1495 run_step  **
           outputs = model.predict_step(data)
       /home/ye53nis/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py:1468 predict_step
           return self(x, training=False)
       /home/ye53nis/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py:1018 __call__
           input_spec.assert_input_compatibility(self.input_spec, inputs, self.name)
       /home/ye53nis/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/keras/engine/input_spec.py:271 assert_input_compatibility
           raise ValueError('Input ' + str(input_index) +
   
       ValueError: Input 0 is incompatible with layer model: expected shape=(None, 16384, 1), found shape=(None, 8192, 1)
   

   So how do we solve this? I did not find the real cause, but a workaround is, to let the loaded model first predict a dummy input with the training size:

   test_features = np.zeros((2**14))
   test_features = np.reshape(test_features, (1, -1, 1))
   print(test_features.shape)
   predictions = model_keras.predict(test_features, verbose=0)
   predictions = predictions.flatten()
   predictions
   

   (1, 16384, 1)
   array([2.7520498e-08, 7.7272375e-09, 5.2612198e-10, ..., 9.8416812e-08,
          8.9488267e-08, 3.5880134e-07], dtype=float32)
   

2. Test the algorithm: here it can be seen, that in the ideal case, the metadata from the single files is joined to each computation. For the computation of all traces, this will probably not be possible, because of traces, which get omitted before correlation, because their trace lengths shrank below 32 time steps after correction. This small traces can't be handled by multipletau.

   path_tb_pex5_egfp = '/beegfs/ye53nis/data/Pablo_structured_experiment'
   pred_thresh = [0.1, 0.3, 0.5, 0.7, 0.9]
   length_delimiter = 2**13  # for U-Net
   bin_for_correlation = 1e5
   out = correction.correct_experimental_traces_from_ptu_by_unet_prediction(
     path_list=path_tb_pex5_egfp,
     model=model_keras,
     pred_thresh=pred_thresh,
     photon_count_bin=bin_for_correlation,
     ntraces=None,
     save_as_csv=True)
   out
   

   Loading dataset 1 from path /beegfs/ye53nis/data/Pablo_structured_experiment with bin=1e6. This can take a while...
   1 of 2: /beegfs/ye53nis/data/Pablo_structured_experiment/DiO LUV 10uM in 20 nM AF48822_T273s_1.ptu
   2 of 2: /beegfs/ye53nis/data/Pablo_structured_experiment/DiO LUV 10uM in 20 nM AF48821_T260s_1.ptu
   Different binning was chosen for correlation. Loading dataset 1 with bin=100000.0. This can take a while...
   Processing correlation of unprocessed dataset 1
   Processing correlation with correction by prediction of dataset 1
   

   	\(D\) in \(\frac{{\mu m\^2}}{{s}}\)	\(\tau\_{{D}}\) in \(ms\)	Trace lengths	folder\id-traces\used	Photon count bin for correlation in \(ns\)	File\GUID	File\CreatingTime	Measurement\SubMode	File\Comment	TTResult\StopReason	…	MeasDesc\GlobalResolution	TTResult\NumberOfRecords	MeasDesc\AcquisitionTime	TTResult\MDescWarningFlags	TTResult\StopAfter	TTResultFormat\TTTRRecType	TTResultFormat\BitsPerRecord	UsrPowerDiode	Header\End	Number of Channels
   0	0.87156	12.932046	8192.0	0-orig	100000.0	{1FADBD63-E853-4886-82EB-0D4FEB0A2641}	(2019, 10, 31, 11, 17, 29, 3, 304, 0)	0		0	…	0.0	6124808	10000	0	10000	16843524	32	1002.253351	<empty Tag>	1
   1	0.091734	122.866229	8192.0	0-orig	100000.0	{1104778F-509F-40AB-9176-580B3A1CE38D}	(2019, 10, 31, 11, 17, 16, 3, 304, 0)	0		0	…	0.0	5881858	10000	0	10000	16843524	32	979.946455	<empty Tag>	1
   2	3.181256	3.542957	6413.0	0-pred-0.1	100000.0	{1FADBD63-E853-4886-82EB-0D4FEB0A2641}	(2019, 10, 31, 11, 17, 29, 3, 304, 0)	0		0	…	0.0	6124808	10000	0	10000	16843524	32	1002.253351	<empty Tag>	1
   3	1.968434	5.725899	7132.0	0-pred-0.1	100000.0	{1104778F-509F-40AB-9176-580B3A1CE38D}	(2019, 10, 31, 11, 17, 16, 3, 304, 0)	0		0	…	0.0	5881858	10000	0	10000	16843524	32	979.946455	<empty Tag>	1
   4	0.534267	21.096303	6903.0	0-pred-0.3	100000.0	{1FADBD63-E853-4886-82EB-0D4FEB0A2641}	(2019, 10, 31, 11, 17, 29, 3, 304, 0)	0		0	…	0.0	6124808	10000	0	10000	16843524	32	1002.253351	<empty Tag>	1
   5	0.760655	14.817573	7490.0	0-pred-0.3	100000.0	{1104778F-509F-40AB-9176-580B3A1CE38D}	(2019, 10, 31, 11, 17, 16, 3, 304, 0)	0		0	…	0.0	5881858	10000	0	10000	16843524	32	979.946455	<empty Tag>	1
   6	0.219843	51.268582	7178.0	0-pred-0.5	100000.0	{1FADBD63-E853-4886-82EB-0D4FEB0A2641}	(2019, 10, 31, 11, 17, 29, 3, 304, 0)	0		0	…	0.0	6124808	10000	0	10000	16843524	32	1002.253351	<empty Tag>	1
   7	0.453877	24.832866	7659.0	0-pred-0.5	100000.0	{1104778F-509F-40AB-9176-580B3A1CE38D}	(2019, 10, 31, 11, 17, 16, 3, 304, 0)	0		0	…	0.0	5881858	10000	0	10000	16843524	32	979.946455	<empty Tag>	1
   8	0.144134	78.198714	7407.0	0-pred-0.7	100000.0	{1FADBD63-E853-4886-82EB-0D4FEB0A2641}	(2019, 10, 31, 11, 17, 29, 3, 304, 0)	0		0	…	0.0	6124808	10000	0	10000	16843524	32	1002.253351	<empty Tag>	1
   9	0.350758	32.133387	7785.0	0-pred-0.7	100000.0	{1104778F-509F-40AB-9176-580B3A1CE38D}	(2019, 10, 31, 11, 17, 16, 3, 304, 0)	0		0	…	0.0	5881858	10000	0	10000	16843524	32	979.946455	<empty Tag>	1
   10	0.092882	121.348056	7692.0	0-pred-0.9	100000.0	{1FADBD63-E853-4886-82EB-0D4FEB0A2641}	(2019, 10, 31, 11, 17, 29, 3, 304, 0)	0		0	…	0.0	6124808	10000	0	10000	16843524	32	1002.253351	<empty Tag>	1
   11	0.281531	40.03487	7937.0	0-pred-0.9	100000.0	{1104778F-509F-40AB-9176-580B3A1CE38D}	(2019, 10, 31, 11, 17, 16, 3, 304, 0)	0		0	…	0.0	5881858	10000	0	10000	16843524	32	979.946455	<empty Tag>	1

   12 rows × 94 columns

3. Run correction for all experimental traces

   path_pex5_exp = ['/beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu', '/beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu']
   pred_thresh = [0.1, 0.3, 0.5, 0.7, 0.9]
   length_delimiter = 2**13  # for U-Net
   bin_for_correlation = 1e5
   out = correction.correct_experimental_traces_from_ptu_by_unet_prediction(
     path_list=path_pex5_exp,
     model=model_keras,
     pred_thresh=pred_thresh,
     photon_count_bin=bin_for_correlation,
     ntraces=400,
     save_as_csv=True)
   out
   

   Loading dataset 1 from path /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu with bin=1e6. This can take a while...
   1 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48891_T1082s_1.ptu
   2 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488129_T1537s_1.ptu
   3 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488210_T2505s_1.ptu
   4 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488199_T2375s_1.ptu
   5 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488171_T2040s_1.ptu
   6 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488229_T2733s_1.ptu
   7 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48890_T1070s_1.ptu
   8 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488113_T1352s_1.ptu
   9 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48883_T986s_1.ptu10 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488227_T2709s_1.ptu
   11 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488174_T2088s_1.ptu
   12 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488102_T1214s_1.ptu
   13 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488131_T1561s_1.ptu
   14 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48826_T302s_1.ptu
   15 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488162_T1932s_1.ptu
   16 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488190_T2267s_1.ptu
   17 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488189_T2255s_1.ptu
   18 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488208_T2481s_1.ptu
   19 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48827_T314s_1.ptu
   20 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488221_T2637s_1.ptu
   21 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488153_T1834s_1.ptu
   22 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488157_T1883s_1.ptu
   23 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488197_T2351s_1.ptu
   24 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48822_T254s_1.ptu
   25 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48816_T182s_1.ptu
   26 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488179_T2148s_1.ptu
   27 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488104_T1238s_1.ptu
   28 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48870_T833s_1.ptu
   29 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48857_T676s_1.ptu
   30 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488125_T1496s_1.ptu
   31 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48839_T459s_1.ptu
   32 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488135_T1610s_1.ptu
   33 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48850_T592s_1.ptu
   34 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488159_T1907s_1.ptu
   35 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48851_T604s_1.ptu
   36 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488158_T1895s_1.ptu
   37 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48878_T926s_1.ptu
   38 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48838_T447s_1.ptu
   39 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48869_T821s_1.ptu
   40 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48834_T399s_1.ptu
   41 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48843_T506s_1.ptu
   42 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48890_T1076s_1.ptu
   43 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488184_T2209s_1.ptu
   44 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48863_T748s_1.ptu
   45 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488103_T1232s_1.ptu
   46 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488181_T2159s_1.ptu
   47 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48859_T698s_1.ptu
   48 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48872_T855s_1.ptu
   49 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488150_T1789s_1.ptu
   50 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488241_T2877s_1.ptu
   51 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488204_T2434s_1.ptu
   52 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48877_T918s_1.ptu
   53 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488180_T2147s_1.ptu
   54 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488183_T2197s_1.ptu
   55 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488160_T1908s_1.ptu
   56 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488238_T2842s_1.ptu
   57 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488165_T1968s_1.ptu
   58 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488234_T2793s_1.ptu
   59 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48882_T979s_1.ptu
   60 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48896_T1148s_1.ptu
   61 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488107_T1280s_1.ptu
   62 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488138_T1654s_1.ptu
   63 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488124_T1484s_1.ptu
   64 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488187_T2245s_1.ptu
   65 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48874_T879s_1.ptu
   66 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488159_T1896s_1.ptu
   67 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488177_T2124s_1.ptu
   68 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488111_T1328s_1.ptu
   69 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48873_T867s_1.ptu
   70 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48881_T962s_1.ptu
   71 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48853_T628s_1.ptu
   72 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48842_T496s_1.ptu
   73 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48850_T590s_1.ptu
   74 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48899_T1184s_1.ptu
   75 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488196_T2339s_1.ptu
   76 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488164_T1956s_1.ptu
   77 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488217_T2589s_1.ptu
   78 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488218_T2601s_1.ptu
   79 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488119_T1418s_1.ptu
   80 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488206_T2458s_1.ptu
   81 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48866_T785s_1.ptu
   82 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488103_T1226s_1.ptu
   83 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48824_T278s_1.ptu
   84 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF4887_T74s_1.ptu
   85 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488176_T2099s_1.ptu
   86 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48817_T194s_1.ptu
   87 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488113_T1346s_1.ptu
   88 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48858_T688s_1.ptu
   89 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488211_T2517s_1.ptu
   90 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48860_T712s_1.ptu
   91 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48833_T387s_1.ptu
   92 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48858_T686s_1.ptu
   93 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48813_T146s_1.ptu
   94 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488126_T1502s_1.ptu
   95 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48894_T1118s_1.ptu
   96 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488110_T1316s_1.ptu
   97 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488188_T2243s_1.ptu
   98 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488120_T1430s_1.ptu
   99 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488219_T2613s_1.ptu
   100 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488138_T1646s_1.ptu
   101 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488187_T2231s_1.ptu
   102 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488146_T1742s_1.ptu
   103 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488155_T1859s_1.ptu
   104 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488174_T2075s_1.ptu
   105 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488158_T1884s_1.ptu
   106 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488244_T2914s_1.ptu
   107 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF4886_T62s_1.ptu
   108 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48885_T1015s_1.ptu
   109 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488165_T1979s_1.ptu
   110 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48857_T674s_1.ptu
   111 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488191_T2279s_1.ptu
   112 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488183_T2183s_1.ptu
   113 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488141_T1689s_1.ptu
   114 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488189_T2269s_1.ptu
   115 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48823_T266s_1.ptu
   116 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488209_T2493s_1.ptu
   117 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488136_T1630s_1.ptu
   118 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488185_T2221s_1.ptu
   119 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488179_T2135s_1.ptu
   120 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48866_T782s_1.ptu
   121 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488123_T1472s_1.ptu
   122 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488160_T1919s_1.ptu
   123 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48856_T664s_1.ptu
   124 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48830_T351s_1.ptu
   125 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48832_T375s_1.ptu
   126 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48859_T700s_1.ptu
   127 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488129_T1545s_1.ptu
   128 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488143_T1714s_1.ptu
   129 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48886_T1022s_1.ptu
   130 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488147_T1754s_1.ptu
   131 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488193_T2303s_1.ptu
   132 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488105_T1250s_1.ptu
   133 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488228_T2721s_1.ptu
   134 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48891_T1088s_1.ptu
   135 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488170_T2040s_1.ptu
   136 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48835_T411s_1.ptu
   137 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48888_T1046s_1.ptu
   138 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488123_T1466s_1.ptu
   139 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48876_T906s_1.ptu
   140 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48822_T255s_1.ptu
   141 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488145_T1730s_1.ptu
   142 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488114_T1358s_1.ptu
   143 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48824_T279s_1.ptu
   144 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48854_T638s_1.ptu
   145 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488247_T2949s_1.ptu
   146 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48844_T519s_1.ptu
   147 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48841_T482s_1.ptu
   148 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48828_T327s_1.ptu
   149 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48840_T471s_1.ptu
   150 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488225_T2685s_1.ptu
   151 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48831_T362s_1.ptu
   152 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48849_T578s_1.ptu
   153 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48861_T722s_1.ptu
   154 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48852_T614s_1.ptu
   155 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48852_T616s_1.ptu
   156 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488149_T1778s_1.ptu
   157 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488169_T2028s_1.ptu
   158 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF4883_T26s_1.ptu
   159 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48881_T966s_1.ptu
   160 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48837_T435s_1.ptu
   161 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488139_T1666s_1.ptu
   162 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488194_T2315s_1.ptu
   163 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488205_T2446s_1.ptu
   164 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48818_T206s_1.ptu
   165 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488128_T1533s_1.ptu
   166 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48851_T602s_1.ptu
   167 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488116_T1388s_1.ptu
   168 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488184_T2196s_1.ptu
   169 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488117_T1394s_1.ptu
   170 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48869_T818s_1.ptu
   171 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48895_T1130s_1.ptu
   172 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488173_T2076s_1.ptu
   173 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488106_T1262s_1.ptu
   174 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488249_T2993s_1.ptu
   175 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48889_T1063s_1.ptu
   176 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48845_T531s_1.ptu
   177 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488117_T1400s_1.ptu
   178 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488192_T2291s_1.ptu
   179 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488166_T1992s_1.ptu
   180 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48896_T1142s_1.ptu
   181 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48894_T1124s_1.ptu
   182 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488177_T2111s_1.ptu
   183 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48862_T736s_1.ptu
   184 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488118_T1412s_1.ptu
   185 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48892_T1100s_1.ptu
   186 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48847_T555s_1.ptu
   187 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488223_T2661s_1.ptu
   188 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48853_T626s_1.ptu
   189 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488136_T1622s_1.ptu
   190 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488151_T1810s_1.ptu
   191 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488245_T2925s_1.ptu
   192 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48879_T942s_1.ptu
   193 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488121_T1448s_1.ptu
   194 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48868_T809s_1.ptu
   195 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488201_T2398s_1.ptu
   196 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488140_T1670s_1.ptu
   197 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488250_T3006s_1.ptu
   198 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF4884_T38s_1.ptu
   199 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488127_T1514s_1.ptu
   200 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488104_T1244s_1.ptu
   201 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488181_T2172s_1.ptu
   202 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488115_T1370s_1.ptu
   203 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488140_T1678s_1.ptu
   204 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48855_T652s_1.ptu
   205 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48884_T1003s_1.ptu
   206 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48846_T544s_1.ptu
   207 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488131_T1569s_1.ptu
   208 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488236_T2817s_1.ptu
   209 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488178_T2123s_1.ptu
   210 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48873_T869s_1.ptu
   211 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488164_T1968s_1.ptu
   212 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488249_T2973s_1.ptu
   213 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488151_T1801s_1.ptu
   214 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488235_T2805s_1.ptu
   215 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488170_T2028s_1.ptu
   216 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488180_T2160s_1.ptu
   217 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488135_T1618s_1.ptu
   218 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488190_T2281s_1.ptu
   219 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48864_T758s_1.ptu
   220 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48867_T794s_1.ptu
   221 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48875_T891s_1.ptu
   222 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48871_T842s_1.ptu
   223 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488213_T2541s_1.ptu
   224 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488182_T2184s_1.ptu
   225 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48841_T484s_1.ptu
   226 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488171_T2052s_1.ptu
   227 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488153_T1825s_1.ptu
   228 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488132_T1581s_1.ptu
   229 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488147_T1762s_1.ptu
   230 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488110_T1310s_1.ptu
   231 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48885_T1010s_1.ptu
   232 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48825_T290s_1.ptu
   233 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488168_T2004s_1.ptu
   234 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48820_T230s_1.ptu
   235 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488162_T1944s_1.ptu
   236 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488203_T2422s_1.ptu
   237 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488152_T1822s_1.ptu
   238 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48821_T243s_1.ptu
   239 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488167_T2004s_1.ptu
   240 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48837_T436s_1.ptu
   241 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488166_T1980s_1.ptu
   242 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488200_T2386s_1.ptu
   243 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488122_T1454s_1.ptu
   244 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48843_T508s_1.ptu
   245 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488102_T1220s_1.ptu
   246 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48865_T773s_1.ptu
   247 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488161_T1920s_1.ptu
   248 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48823_T267s_1.ptu
   249 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48863_T746s_1.ptu
   250 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488132_T1573s_1.ptu
   251 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48867_T797s_1.ptu
   252 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488144_T1726s_1.ptu
   253 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488145_T1738s_1.ptu
   254 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488169_T2016s_1.ptu
   255 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48889_T1058s_1.ptu
   256 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48860_T710s_1.ptu
   257 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48831_T363s_1.ptu
   258 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488215_T2565s_1.ptu
   259 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48878_T930s_1.ptu
   260 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48865_T770s_1.ptu
   261 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48870_T830s_1.ptu
   262 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488114_T1364s_1.ptu
   263 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488148_T1774s_1.ptu
   264 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488224_T2673s_1.ptu
   265 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488172_T2052s_1.ptu
   266 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488167_T1992s_1.ptu
   267 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488100_T1190s_1.ptu
   268 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48821_T242s_1.ptu
   269 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488240_T2865s_1.ptu
   270 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488142_T1701s_1.ptu
   271 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488130_T1549s_1.ptu
   272 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48844_T520s_1.ptu
   273 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48861_T724s_1.ptu
   274 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48814_T158s_1.ptu
   275 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488163_T1956s_1.ptu
   276 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488173_T2064s_1.ptu
   277 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF4885_T50s_1.ptu
   278 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48849_T580s_1.ptu
   279 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48840_T472s_1.ptu
   280 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48883_T991s_1.ptu
   281 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488247_T2969s_1.ptu
   282 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488150_T1798s_1.ptu
   283 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48848_T566s_1.ptu
   284 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48897_T1160s_1.ptu
   285 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488118_T1406s_1.ptu
   286 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF4889_T98s_1.ptu
   287 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48812_T134s_1.ptu
   288 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488222_T2649s_1.ptu
   289 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48887_T1039s_1.ptu
   290 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48879_T938s_1.ptu
   291 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48888_T1051s_1.ptu
   292 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48876_T902s_1.ptu
   293 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48819_T218s_1.ptu
   294 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48893_T1112s_1.ptu
   295 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488121_T1442s_1.ptu
   296 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488101_T1208s_1.ptu
   297 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488109_T1304s_1.ptu
   298 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488128_T1526s_1.ptu
   299 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488207_T2470s_1.ptu
   300 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488246_T2937s_1.ptu
   301 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488242_T2890s_1.ptu
   302 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48854_T640s_1.ptu
   303 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48847_T554s_1.ptu
   304 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488108_T1292s_1.ptu
   305 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488134_T1597s_1.ptu
   306 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF4888_T86s_1.ptu
   307 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48872_T857s_1.ptu
   308 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488149_T1786s_1.ptu
   309 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48825_T291s_1.ptu
   310 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488182_T2171s_1.ptu
   311 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488156_T1871s_1.ptu
   312 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488220_T2624s_1.ptu
   313 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488248_T2981s_1.ptu
   314 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48897_T1154s_1.ptu
   315 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48848_T568s_1.ptu
   316 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488175_T2100s_1.ptu
   317 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488154_T1847s_1.ptu
   318 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488245_T2945s_1.ptu
   319 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488139_T1658s_1.ptu
   320 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48830_T350s_1.ptu
   321 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488175_T2087s_1.ptu
   322 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48856_T662s_1.ptu
   323 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488230_T2745s_1.ptu
   324 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48875_T894s_1.ptu
   325 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488127_T1521s_1.ptu
   326 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488195_T2327s_1.ptu
   327 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48884_T998s_1.ptu
   328 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488186_T2219s_1.ptu
   329 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48882_T974s_1.ptu
   330 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488134_T1606s_1.ptu
   331 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488108_T1286s_1.ptu
   332 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48827_T315s_1.ptu
   333 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488214_T2553s_1.ptu
   334 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488163_T1944s_1.ptu
   335 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48839_T460s_1.ptu
   336 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488186_T2233s_1.ptu
   337 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488243_T2902s_1.ptu
   338 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488137_T1642s_1.ptu
   339 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488156_T1861s_1.ptu
   340 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF4882_T14s_1.ptu
   341 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488142_T1694s_1.ptu
   342 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48811_T123s_1.ptu
   343 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488248_T2961s_1.ptu
   344 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48895_T1136s_1.ptu
   345 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48826_T303s_1.ptu
   346 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488191_T2293s_1.ptu
   347 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488137_T1634s_1.ptu
   348 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488133_T1585s_1.ptu
   349 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488216_T2577s_1.ptu
   350 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488107_T1274s_1.ptu
   351 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488130_T1557s_1.ptu
   352 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488120_T1436s_1.ptu
   353 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48868_T806s_1.ptu
   354 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488125_T1490s_1.ptu
   355 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48832_T374s_1.ptu
   356 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488119_T1424s_1.ptu
   357 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48877_T914s_1.ptu
   358 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488188_T2257s_1.ptu
   359 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48836_T423s_1.ptu
   360 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488176_T2112s_1.ptu
   361 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488112_T1334s_1.ptu
   362 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48815_T170s_1.ptu
   363 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48838_T448s_1.ptu
   364 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488178_T2136s_1.ptu
   365 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488198_T2363s_1.ptu
   366 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48855_T650s_1.ptu
   367 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48899_T1178s_1.ptu
   368 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488161_T1931s_1.ptu
   369 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488246_T2957s_1.ptu
   370 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488231_T2757s_1.ptu
   371 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488122_T1460s_1.ptu
   372 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488157_T1873s_1.ptu
   373 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488194_T2329s_1.ptu
   374 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48846_T542s_1.ptu
   375 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488144_T1718s_1.ptu
   376 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488111_T1322s_1.ptu
   377 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488143_T1705s_1.ptu
   378 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48874_T882s_1.ptu
   379 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488124_T1478s_1.ptu
   380 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488106_T1268s_1.ptu
   381 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48892_T1094s_1.ptu
   382 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488115_T1376s_1.ptu
   383 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48864_T761s_1.ptu
   384 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488239_T2853s_1.ptu
   385 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48893_T1106s_1.ptu
   386 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488154_T1837s_1.ptu
   387 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488105_T1256s_1.ptu
   388 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48829_T338s_1.ptu
   389 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48871_T845s_1.ptu
   390 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48898_T1172s_1.ptu
   391 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48880_T954s_1.ptu
   392 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48811_T122s_1.ptu
   393 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488141_T1682s_1.ptu
   394 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488237_T2829s_1.ptu
   395 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48810_T110s_1.ptu
   396 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48898_T1166s_1.ptu
   397 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48812_T135s_1.ptu
   398 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48887_T1034s_1.ptu
   399 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488202_T2410s_1.ptu
   400 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48828_T326s_1.ptu
   401 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48886_T1027s_1.ptu
   Different binning was chosen for correlation. Loading dataset 1 with bin=100000.0. This can take a while...
   Processing correlation of unprocessed dataset 1
   Processing correlation with correction by prediction of dataset 1
   Loading dataset 2 from path /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu with bin=1e6. This can take a while...
   1 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48869_T825s_1.ptu
   2 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488196_T2359s_1.ptu
   3 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488146_T1755s_1.ptu
   4 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488227_T2733s_1.ptu
   5 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48823_T287s_1.ptu
   6 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48829_T342s_1.ptu
   7 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488194_T2334s_1.ptu
   8 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48851_T607s_1.ptu
   9 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488210_T2528s_1.ptu
   10 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488234_T2818s_1.ptu
   11 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48882_T982s_1.ptu
   12 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488251_T3022s_1.ptu
   13 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48848_T610s_1.ptu
   14 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488160_T1924s_1.ptu
   15 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48846_T584s_1.ptu
   16 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488268_T3227s_1.ptu
   17 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48823_T268s_1.ptu
   18 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48832_T378s_1.ptu
   19 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488241_T2902s_1.ptu
   20 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488198_T2383s_1.ptu
   21 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48869_T823s_1.ptu
   22 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48895_T1139s_1.ptu
   23 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48878_T934s_1.ptu
   24 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488100_T1197s_1.ptu
   25 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488155_T1864s_1.ptu
   26 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488132_T1585s_1.ptu
   27 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48866_T787s_1.ptu
   28 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48877_T922s_1.ptu
   29 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48861_T728s_1.ptu
   30 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488129_T1548s_1.ptu
   31 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48825_T292s_1.ptu
   32 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488205_T2468s_1.ptu
   33 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4889_T106s_1.ptu
   34 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48840_T474s_1.ptu
   35 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48843_T546s_1.ptu
   36 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48887_T1042s_1.ptu
   37 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488116_T1391s_1.ptu
   38 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4884_T39s_1.ptu
   39 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488110_T1318s_1.ptu
   40 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488218_T2625s_1.ptu
   41 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48870_T834s_1.ptu
   42 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488270_T3251s_1.ptu
   43 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488177_T2129s_1.ptu
   44 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488136_T1633s_1.ptu
   45 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48844_T521s_1.ptu
   46 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48829_T341s_1.ptu
   47 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4887_T75s_1.ptu
   48 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48842_T497s_1.ptu
   49 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488291_T3505s_1.ptu
   50 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488117_T1403s_1.ptu
   51 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48814_T159s_1.ptu
   52 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488215_T2589s_1.ptu
   53 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488128_T1536s_1.ptu
   54 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488135_T1621s_1.ptu
   55 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48870_T837s_1.ptu
   56 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488239_T2878s_1.ptu
   57 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48827_T317s_1.ptu
   58 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488231_T2781s_1.ptu
   59 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488294_T3541s_1.ptu
   60 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488247_T2974s_1.ptu
   61 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488226_T2721s_1.ptu
   62 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488182_T2189s_1.ptu
   63 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488248_T2986s_1.ptu
   64 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488216_T2601s_1.ptu
   65 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48824_T282s_1.ptu
   66 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488152_T1827s_1.ptu
   67 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48868_T811s_1.ptu
   68 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488112_T1342s_1.ptu
   69 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48811_T123s_1.ptu
   70 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488185_T2225s_1.ptu
   71 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48828_T330s_1.ptu
   72 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488232_T2793s_1.ptu
   73 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488293_T3529s_1.ptu
   74 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48825_T294s_1.ptu
   75 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4886_T66s_1.ptu
   76 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48822_T273s_1.ptu
   77 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48889_T1067s_1.ptu
   78 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48816_T183s_1.ptu
   79 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48879_T946s_1.ptu
   80 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48876_T910s_1.ptu
   81 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48840_T507s_1.ptu
   82 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48849_T583s_1.ptu
   83 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48850_T595s_1.ptu
   84 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48822_T257s_1.ptu
   85 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488184_T2213s_1.ptu
   86 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48813_T148s_1.ptu
   87 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488121_T1452s_1.ptu
   88 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488254_T3059s_1.ptu
   89 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48814_T171s_1.ptu
   90 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48849_T581s_1.ptu
   91 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488122_T1464s_1.ptu
   92 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488153_T1839s_1.ptu
   93 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48851_T605s_1.ptu
   94 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48819_T222s_1.ptu
   95 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488104_T1246s_1.ptu
   96 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488179_T2153s_1.ptu
   97 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488130_T1560s_1.ptu
   98 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48874_T885s_1.ptu
   99 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488115_T1379s_1.ptu
   100 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488282_T3397s_1.ptu
   101 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488259_T3119s_1.ptu
   102 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48858_T690s_1.ptu
   103 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48821_T244s_1.ptu
   104 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48825_T313s_1.ptu
   105 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48861_T726s_1.ptu
   106 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48817_T196s_1.ptu
   107 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488265_T3191s_1.ptu
   108 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48834_T400s_1.ptu
   109 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48854_T643s_1.ptu
   110 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48873_T873s_1.ptu
   111 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48857_T678s_1.ptu
   112 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488208_T2504s_1.ptu
   113 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4884_T41s_1.ptu
   114 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48810_T120s_1.ptu
   115 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48873_T870s_1.ptu
   116 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48872_T859s_1.ptu
   117 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488180_T2165s_1.ptu
   118 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48828_T328s_1.ptu
   119 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488224_T2697s_1.ptu
   120 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488167_T2009s_1.ptu
   121 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488237_T2854s_1.ptu
   122 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488258_T3107s_1.ptu
   123 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488222_T2673s_1.ptu
   124 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48845_T534s_1.ptu
   125 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488280_T3373s_1.ptu
   126 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48848_T569s_1.ptu
   127 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48812_T145s_1.ptu
   128 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48844_T522s_1.ptu
   129 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48828_T351s_1.ptu
   130 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488173_T2081s_1.ptu
   131 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48893_T1115s_1.ptu
   132 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48865_T774s_1.ptu
   133 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48862_T740s_1.ptu
   134 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488187_T2250s_1.ptu
   135 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48812_T136s_1.ptu
   136 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48853_T631s_1.ptu
   137 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48843_T509s_1.ptu
   138 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48880_T955s_1.ptu
   139 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48845_T535s_1.ptu
   140 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488133_T1597s_1.ptu
   141 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48836_T426s_1.ptu
   142 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488138_T1658s_1.ptu
   143 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488235_T2830s_1.ptu
   144 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488111_T1330s_1.ptu
   145 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488156_T1876s_1.ptu
   146 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48855_T655s_1.ptu
   147 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4887_T80s_1.ptu
   148 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48831_T365s_1.ptu
   149 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48854_T642s_1.ptu
   150 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48872_T861s_1.ptu
   151 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488107_T1282s_1.ptu
   152 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48824_T280s_1.ptu
   153 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488256_T3083s_1.ptu
   154 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488204_T2456s_1.ptu
   155 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488197_T2371s_1.ptu
   156 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48852_T619s_1.ptu
   157 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488176_T2117s_1.ptu
   158 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488229_T2757s_1.ptu
   159 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48842_T498s_1.ptu
   160 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48826_T305s_1.ptu
   161 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48864_T762s_1.ptu
   162 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488289_T3481s_1.ptu
   163 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48849_T624s_1.ptu
   164 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488249_T2998s_1.ptu
   165 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48816_T184s_1.ptu
   166 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48817_T197s_1.ptu
   167 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48884_T1006s_1.ptu
   168 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488178_T2141s_1.ptu
   169 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48855_T654s_1.ptu
   170 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48813_T158s_1.ptu
   171 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488118_T1415s_1.ptu
   172 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488127_T1524s_1.ptu
   173 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48836_T425s_1.ptu
   174 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48830_T352s_1.ptu
   175 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48829_T340s_1.ptu
   176 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488261_T3143s_1.ptu
   177 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48837_T468s_1.ptu
   178 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48830_T354s_1.ptu
   179 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48819_T220s_1.ptu
   180 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488236_T2842s_1.ptu
   181 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488257_T3095s_1.ptu
   182 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48819_T219s_1.ptu
   183 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488253_T3047s_1.ptu
   184 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48841_T485s_1.ptu
   185 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48830_T377s_1.ptu
   186 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488201_T2420s_1.ptu
   187 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48835_T412s_1.ptu
   188 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488269_T3239s_1.ptu
   189 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48846_T545s_1.ptu
   190 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48833_T416s_1.ptu
   191 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48839_T461s_1.ptu
   192 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48850_T637s_1.ptu
   193 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488276_T3324s_1.ptu
   194 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488286_T3445s_1.ptu
   195 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488181_T2177s_1.ptu
   196 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488199_T2395s_1.ptu
   197 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488244_T2938s_1.ptu
   198 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488101_T1209s_1.ptu
   199 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48881_T970s_1.ptu
   200 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488126_T1512s_1.ptu
   201 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488223_T2685s_1.ptu
   202 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48847_T557s_1.ptu
   203 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488158_T1900s_1.ptu
   204 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488139_T1670s_1.ptu
   205 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48824_T281s_1.ptu
   206 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48859_T702s_1.ptu
   207 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48846_T546s_1.ptu
   208 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488267_T3215s_1.ptu
   209 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4883_T28s_1.ptu
   210 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48827_T318s_1.ptu
   211 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48820_T232s_1.ptu
   212 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48860_T716s_1.ptu
   213 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4882_T15s_1.ptu
   214 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48896_T1151s_1.ptu
   215 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48830_T353s_1.ptu
   216 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488262_T3155s_1.ptu
   217 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48899_T1187s_1.ptu
   218 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488209_T2516s_1.ptu
   219 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488150_T1803s_1.ptu
   220 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488147_T1767s_1.ptu
   221 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488134_T1609s_1.ptu
   222 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48815_T171s_1.ptu
   223 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488264_T3179s_1.ptu
   224 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488298_T3590s_1.ptu
   225 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488193_T2322s_1.ptu
   226 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4888_T87s_1.ptu
   227 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488106_T1270s_1.ptu
   228 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488284_T3421s_1.ptu
   229 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488145_T1743s_1.ptu
   230 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48828_T329s_1.ptu
   231 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488183_T2201s_1.ptu
   232 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48856_T668s_1.ptu
   233 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4888_T93s_1.ptu
   234 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4889_T100s_1.ptu
   235 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48818_T222s_1.ptu
   236 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48835_T442s_1.ptu
   237 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488175_T2105s_1.ptu
   238 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48841_T486s_1.ptu
   239 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48833_T390s_1.ptu
   240 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488245_T2950s_1.ptu
   241 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488195_T2347s_1.ptu
   242 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48867_T799s_1.ptu
   243 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48839_T494s_1.ptu
   244 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48817_T195s_1.ptu
   245 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488105_T1258s_1.ptu
   246 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48897_T1163s_1.ptu
   247 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48835_T413s_1.ptu
   248 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48820_T248s_1.ptu
   249 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48841_T520s_1.ptu
   250 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4881_T0s_1.ptu
   251 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488165_T1984s_1.ptu
   252 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48887_T1040s_1.ptu
   253 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48817_T210s_1.ptu
   254 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4883_T27s_1.ptu
   255 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48891_T1091s_1.ptu
   256 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488233_T2806s_1.ptu
   257 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48811_T132s_1.ptu
   258 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48876_T906s_1.ptu
   259 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48839_T462s_1.ptu
   260 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488109_T1306s_1.ptu
   261 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488162_T1948s_1.ptu
   262 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48858_T692s_1.ptu
   263 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488151_T1815s_1.ptu
   264 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488203_T2444s_1.ptu
   265 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48815_T184s_1.ptu
   266 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48831_T366s_1.ptu
   267 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48883_T991s_1.ptu
   268 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48881_T967s_1.ptu
   269 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488287_T3457s_1.ptu
   270 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48884_T1003s_1.ptu
   271 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48886_T1031s_1.ptu
   272 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48820_T234s_1.ptu
   273 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48845_T533s_1.ptu
   274 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488100_T1199s_1.ptu
   275 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488214_T2577s_1.ptu
   276 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48845_T571s_1.ptu
   277 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48856_T666s_1.ptu
   278 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488172_T2069s_1.ptu
   279 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48837_T438s_1.ptu
   280 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48822_T256s_1.ptu
   281 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488243_T2926s_1.ptu
   282 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488240_T2890s_1.ptu
   283 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488206_T2480s_1.ptu
   284 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48818_T208s_1.ptu
   285 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48866_T788s_1.ptu
   286 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48863_T752s_1.ptu
   287 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488238_T2866s_1.ptu
   288 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488260_T3131s_1.ptu
   289 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48818_T210s_1.ptu
   290 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48892_T1103s_1.ptu
   291 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48813_T147s_1.ptu
   292 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48834_T402s_1.ptu
   293 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48853_T630s_1.ptu
   294 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488114_T1367s_1.ptu
   295 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488300_T3614s_1.ptu
   296 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488119_T1427s_1.ptu
   297 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48868_T813s_1.ptu
   298 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488157_T1888s_1.ptu
   299 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488188_T2262s_1.ptu
   300 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488212_T2552s_1.ptu
   301 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48833_T388s_1.ptu
   302 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48864_T764s_1.ptu
   303 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48823_T269s_1.ptu
   304 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488252_T3035s_1.ptu
   305 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488169_T2032s_1.ptu
   306 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48890_T1079s_1.ptu
   307 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48871_T847s_1.ptu
   308 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488137_T1646s_1.ptu
   309 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488166_T1997s_1.ptu
   310 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488283_T3409s_1.ptu
   311 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488279_T3360s_1.ptu
   312 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488125_T1500s_1.ptu
   313 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488200_T2408s_1.ptu
   314 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48882_T979s_1.ptu
   315 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488281_T3385s_1.ptu
   316 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48811_T124s_1.ptu
   317 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488102_T1221s_1.ptu
   318 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48825_T293s_1.ptu
   319 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488186_T2237s_1.ptu
   320 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488274_T3299s_1.ptu
   321 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48838_T450s_1.ptu
   322 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48826_T304s_1.ptu
   323 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488163_T1960s_1.ptu
   324 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488174_T2093s_1.ptu
   325 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488250_T3010s_1.ptu
   326 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488255_T3071s_1.ptu
   327 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48847_T597s_1.ptu
   328 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488288_T3469s_1.ptu
   329 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488103_T1234s_1.ptu
   330 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488164_T1972s_1.ptu
   331 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48885_T1018s_1.ptu
   332 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48821_T246s_1.ptu
   333 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48812_T135s_1.ptu
   334 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488242_T2914s_1.ptu
   335 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488142_T1706s_1.ptu
   336 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488159_T1912s_1.ptu
   337 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4887_T76s_1.ptu
   338 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48886_T1028s_1.ptu
   339 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488263_T3167s_1.ptu
   340 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48824_T299s_1.ptu
   341 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488113_T1354s_1.ptu
   342 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488108_T1294s_1.ptu
   343 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48816_T197s_1.ptu
   344 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48863_T750s_1.ptu
   345 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488140_T1682s_1.ptu
   346 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488230_T2769s_1.ptu
   347 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488131_T1573s_1.ptu
   348 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488220_T2649s_1.ptu
   349 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48838_T449s_1.ptu
   350 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488154_T1852s_1.ptu
   351 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48838_T481s_1.ptu
   352 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488120_T1439s_1.ptu
   353 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488207_T2492s_1.ptu
   354 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488171_T2056s_1.ptu
   355 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488190_T2286s_1.ptu
   356 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48847_T558s_1.ptu
   357 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48898_T1175s_1.ptu
   358 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48862_T738s_1.ptu
   359 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48816_T186s_1.ptu
   360 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488299_T3602s_1.ptu
   361 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488219_T2637s_1.ptu
   362 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48810_T111s_1.ptu
   363 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488278_T3348s_1.ptu
   364 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488189_T2274s_1.ptu
   365 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48859_T704s_1.ptu
   366 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48832_T376s_1.ptu
   367 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48865_T776s_1.ptu
   368 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488225_T2709s_1.ptu
   369 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48871_T849s_1.ptu
   370 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48883_T994s_1.ptu
   371 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48831_T390s_1.ptu
   372 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488272_T3276s_1.ptu
   373 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488148_T1779s_1.ptu
   374 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48875_T898s_1.ptu
   375 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48842_T532s_1.ptu
   376 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488290_T3493s_1.ptu
   377 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48877_T918s_1.ptu
   378 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488297_T3578s_1.ptu
   379 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488191_T2298s_1.ptu
   380 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48814_T160s_1.ptu
   381 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48844_T558s_1.ptu
   382 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488221_T2661s_1.ptu
   383 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488211_T2540s_1.ptu
   384 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48821_T260s_1.ptu
   385 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488123_T1475s_1.ptu
   386 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48852_T617s_1.ptu
   387 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488149_T1791s_1.ptu
   388 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488141_T1694s_1.ptu
   389 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48827_T338s_1.ptu
   390 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488217_T2613s_1.ptu
   391 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48848_T571s_1.ptu
   392 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4886_T63s_1.ptu
   393 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48843_T510s_1.ptu
   394 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48810_T112s_1.ptu
   395 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488124_T1488s_1.ptu
   396 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488271_T3263s_1.ptu
   397 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48840_T473s_1.ptu
   398 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48826_T306s_1.ptu
   399 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4885_T54s_1.ptu
   400 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48827_T316s_1.ptu
   401 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48850_T593s_1.ptu
   Different binning was chosen for correlation. Loading dataset 2 with bin=100000.0. This can take a while...
   Processing correlation of unprocessed dataset 2
   Processing correlation with correction by prediction of dataset 2
   src/fluotracify/applications/correction.py:508: UserWarning: Metadata is not saved with data. Reason: the correlation algorithm failed for one or more traces which were shorter than 32 time steps after correction.Since metadata is loaded in the beginning, it is not sure, which correlation is missing to ensure proper joining of data and metadata.
     'correlation algorithm failed for one or more traces '
   

   	\(D\) in \(\frac{{\mu m\^2}}{{s}}\)	\(\tau\_{{D}}\) in \(ms\)	Trace lengths	folder\id-traces\used	Photon count bin for correlation in \(ns\)
   0	19.078159	0.590783	8192.0	0-orig	100000.0
   1	20.314897	0.554817	8192.0	0-orig	100000.0
   2	21.007772	0.536518	8192.0	0-orig	100000.0
   3	22.716317	0.496166	8192.0	0-orig	100000.0
   4	24.089236	0.467888	8192.0	0-orig	100000.0
   …	…	…	…	…	…
   4697	0.006707	1680.558197	7719.0	1-pred-0.9	100000.0
   4698	0.071281	158.122526	7592.0	1-pred-0.9	100000.0
   4699	1.228636	9.173635	7876.0	1-pred-0.9	100000.0
   4700	0.258008	43.684844	7830.0	1-pred-0.9	100000.0
   4701	0.436051	25.848023	7974.0	1-pred-0.9	100000.0

   4702 rows × 5 columns

1. current directory

   %cd /beegfs/ye53nis/drmed-git
   

   /beegfs/ye53nis/drmed-git
   

1. git log

   !git log -3
   

   commit fa530079f3d8bdbba1eaa4c6bd3eed6a24789d84
   Author: Apoplex <oligolex@vivaldi.net>
   Date:   Tue Mar 16 22:20:39 2021 +0100
   
       update mlproject for separate train and test files
   
   commit c9dd5f025bbd95e58f8f34796be36bcdb8c1a253
   Author: Apoplex <oligolex@vivaldi.net>
   Date:   Tue Mar 16 22:13:35 2021 +0100
   
       Update train for separate train and test paths
   
   commit a070d3b531725e0fb37688dde80e990083ccf1cc
   Author: Apoplex <oligolex@vivaldi.net>
   Date:   Mon Mar 15 21:48:01 2021 +0100
   
       Fix photon count bin metadata 2
   

2. Metadata of environment

   No of CPUs in system: 72
   No of CPUs the current process can use: 24
   load average: (29.05, 36.29, 43.21)
   os.uname():  posix.uname_result(sysname='Linux', nodename='node260', release='3.10.0-957.1.3.el7.x86_64', version='#1 SMP Thu Nov 29 14:49:43 UTC 2018', machine='x86_64')
   PID of process: 41375
   RAM total: 199G, RAM used: 7.3G, RAM free: 116G
   the current directory: /beegfs/ye53nis/drmed-git
   My disk usage:
   Filesystem           Size  Used Avail Use% Mounted on
   /dev/sda1             50G  3.3G   47G   7% /
   devtmpfs              94G     0   94G   0% /dev
   tmpfs                 94G  133M   94G   1% /dev/shm
   tmpfs                 94G   19M   94G   1% /run
   tmpfs                 94G     0   94G   0% /sys/fs/cgroup
   nfs02-ib:/data01      88T   69T   19T  79% /data01
   nfs01-ib:/home        80T   66T   15T  82% /home
   nfs01-ib:/cluster    2.0T  417G  1.6T  21% /cluster
   nfs03-ib:/pool/work  100T   78T   23T  78% /nfsdata
   /dev/sda5            2.0G   66M  2.0G   4% /tmp
   /dev/sda6            169G  3.0G  166G   2% /local
   /dev/sda3            6.0G  397M  5.7G   7% /var
   beegfs_nodev         524T  303T  222T  58% /beegfs
   tmpfs                 19G     0   19G   0% /run/user/67339# packages in environment at /home/ye53nis/.conda/envs/tf-nightly:
   #
   # Name                    Version                   Build  Channel
   _libgcc_mutex             0.1                        main
   absl-py                   0.11.0                   pypi_0    pypi
   alembic                   1.4.1                      py_0    conda-forge/label/main
   appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge/label/main
   argon2-cffi               20.1.0           py38h7b6447c_1
   asn1crypto                1.4.0              pyh9f0ad1d_0    conda-forge/label/main
   asteval                   0.9.16             pyh5ca1d4c_0    conda-forge/label/main
   astunparse                1.6.3                    pypi_0    pypi
   async_generator           1.10               pyhd3eb1b0_0
   attrs                     20.3.0             pyhd3eb1b0_0
   azure-core                1.10.0             pyhd8ed1ab_0    conda-forge/label/main
   azure-storage-blob        12.7.1             pyh44b312d_0    conda-forge/label/main
   backcall                  0.2.0              pyhd3eb1b0_0
   blas                      1.0                         mkl
   bleach                    3.2.3              pyhd3eb1b0_0
   blinker                   1.4                        py_1    conda-forge/label/main
   blosc                     1.20.1               hd408876_0
   brotli                    1.0.9                he6710b0_2
   brotlipy                  0.7.0           py38h27cfd23_1003
   brunsli                   0.1                  h2531618_0
   bzip2                     1.0.8                h7b6447c_0
   ca-certificates           2020.12.5            ha878542_0    conda-forge/label/main
   cachetools                4.2.1                    pypi_0    pypi
   certifi                   2020.12.5        py38h578d9bd_1    conda-forge/label/main
   cffi                      1.14.4           py38h261ae71_0
   chardet                   4.0.0           py38h06a4308_1003
   charls                    2.1.0                he6710b0_2
   click                     7.1.2              pyh9f0ad1d_0    conda-forge/label/main
   cloudpickle               1.6.0                      py_0    conda-forge/label/main
   configparser              5.0.1                      py_0    conda-forge/label/main
   cryptography              3.3.1            py38h3c74f83_0
   cycler                    0.10.0                   py38_0
   databricks-cli            0.9.1                      py_0    conda-forge/label/main
   dbus                      1.13.18              hb2f20db_0
   decorator                 4.4.2              pyhd3eb1b0_0
   defusedxml                0.6.0                      py_0
   docker-py                 4.4.1            py38h578d9bd_1    conda-forge/label/main
   docker-pycreds            0.4.0                      py_0    conda-forge/label/main
   entrypoints               0.3                      py38_0
   expat                     2.2.10               he6710b0_2
   fcsfiles                  2020.9.18                pypi_0    pypi
   flask                     1.1.2              pyh9f0ad1d_0    conda-forge/label/main
   flatbuffers               1.12                     pypi_0    pypi
   fontconfig                2.13.0               h9420a91_0
   freetype                  2.10.4               h5ab3b9f_0
   future                    0.18.2           py38h578d9bd_3    conda-forge/label/main
   gast                      0.4.0                    pypi_0    pypi
   giflib                    5.1.4                h14c3975_1
   gitdb                     4.0.5                      py_0    conda-forge/label/main
   gitpython                 3.1.12             pyhd8ed1ab_0    conda-forge/label/main
   glib                      2.66.1               h92f7085_0
   google-auth               1.24.0                   pypi_0    pypi
   google-auth-oauthlib      0.4.2                    pypi_0    pypi
   google-pasta              0.2.0                    pypi_0    pypi
   gorilla                   0.3.0                      py_0    conda-forge/label/main
   grpcio                    1.34.1                   pypi_0    pypi
   gst-plugins-base          1.14.0               h8213a91_2
   gstreamer                 1.14.0               h28cd5cc_2
   gunicorn                  20.0.4           py38h578d9bd_3    conda-forge/label/main
   h5py                      3.1.0                    pypi_0    pypi
   icu                       58.2                 he6710b0_3
   idna                      2.10               pyhd3eb1b0_0
   imagecodecs               2021.1.11        py38h581e88b_1
   importlib-metadata        2.0.0                      py_1
   importlib_metadata        2.0.0                         1
   intel-openmp              2020.2                      254
   ipykernel                 5.3.4            py38h5ca1d4c_0
   ipython                   7.19.0           py38hb070fc8_1
   ipython_genutils          0.2.0              pyhd3eb1b0_1
   isodate                   0.6.0                      py_1    conda-forge/label/main
   itsdangerous              1.1.0                      py_0    conda-forge/label/main
   jedi                      0.17.0                   py38_0
   jinja2                    2.11.2             pyhd3eb1b0_0
   jpeg                      9b                   h024ee3a_2
   json5                     0.9.5                      py_0
   jsonschema                3.2.0                      py_2
   jupyter_client            6.1.7                      py_0
   jupyter_core              4.7.0            py38h06a4308_0
   jupyterlab                2.2.6                      py_0
   jupyterlab_pygments       0.1.2                      py_0
   jupyterlab_server         1.2.0                      py_0
   jxrlib                    1.1                  h7b6447c_2
   keras-preprocessing       1.1.2                    pypi_0    pypi
   kiwisolver                1.3.0            py38h2531618_0
   lcms2                     2.11                 h396b838_0
   ld_impl_linux-64          2.33.1               h53a641e_7
   lerc                      2.2.1                h2531618_0
   libaec                    1.0.4                he6710b0_1
   libdeflate                1.7                  h27cfd23_5
   libedit                   3.1.20191231         h14c3975_1
   libffi                    3.3                  he6710b0_2
   libgcc-ng                 9.1.0                hdf63c60_0
   libgfortran-ng            7.3.0                hdf63c60_0
   libpng                    1.6.37               hbc83047_0
   libprotobuf               3.13.0.1             h8b12597_0    conda-forge/label/main
   libsodium                 1.0.18               h7b6447c_0
   libstdcxx-ng              9.1.0                hdf63c60_0
   libtiff                   4.1.0                h2733197_1
   libuuid                   1.0.3                h1bed415_2
   libwebp                   1.0.1                h8e7db2f_0
   libxcb                    1.14                 h7b6447c_0
   libxml2                   2.9.10               hb55368b_3
   libzopfli                 1.0.3                he6710b0_0
   lmfit                     1.0.1                      py_1    conda-forge/label/main
   lz4-c                     1.9.3                h2531618_0
   mako                      1.1.4              pyh44b312d_0    conda-forge/label/main
   markdown                  3.3.3                    pypi_0    pypi
   markupsafe                1.1.1            py38h7b6447c_0
   matplotlib                3.3.2                h06a4308_0
   matplotlib-base           3.3.2            py38h817c723_0
   mistune                   0.8.4           py38h7b6447c_1000
   mkl                       2020.2                      256
   mkl-service               2.3.0            py38he904b0f_0
   mkl_fft                   1.2.0            py38h23d657b_0
   mkl_random                1.1.1            py38h0573a6f_0
   mlflow                    1.13.1           py38h578d9bd_2    conda-forge/label/main
   msrest                    0.6.21             pyh44b312d_0    conda-forge/label/main
   multipletau               0.3.3                    pypi_0    pypi
   nbclient                  0.5.1                      py_0
   nbconvert                 6.0.7                    py38_0
   nbformat                  5.1.2              pyhd3eb1b0_1
   ncurses                   6.2                  he6710b0_1
   nest-asyncio              1.4.3              pyhd3eb1b0_0
   notebook                  6.2.0            py38h06a4308_0
   numpy                     1.19.2           py38h54aff64_0
   numpy-base                1.19.2           py38hfa32c7d_0
   oauthlib                  3.0.1                      py_0    conda-forge/label/main
   olefile                   0.46                       py_0
   openjpeg                  2.3.0                h05c96fa_1
   openssl                   1.1.1i               h27cfd23_0
   opt-einsum                3.3.0                    pypi_0    pypi
   packaging                 20.9               pyhd3eb1b0_0
   pandas                    1.2.1            py38ha9443f7_0
   pandoc                    2.11                 hb0f4dca_0
   pandocfilters             1.4.3            py38h06a4308_1
   parso                     0.8.1              pyhd3eb1b0_0
   pcre                      8.44                 he6710b0_0
   pexpect                   4.8.0              pyhd3eb1b0_3
   pickleshare               0.7.5           pyhd3eb1b0_1003
   pillow                    8.1.0            py38he98fc37_0
   pip                       20.3.3           py38h06a4308_0
   prometheus_client         0.9.0              pyhd3eb1b0_0
   prometheus_flask_exporter 0.18.1             pyh9f0ad1d_0    conda-forge/label/main
   prompt-toolkit            3.0.8                      py_0
   protobuf                  3.13.0.1         py38hadf7658_1    conda-forge/label/main
   ptyprocess                0.7.0              pyhd3eb1b0_2
   pyasn1                    0.4.8                    pypi_0    pypi
   pyasn1-modules            0.2.8                    pypi_0    pypi
   pycparser                 2.20                       py_2
   pygments                  2.7.4              pyhd3eb1b0_0
   pyjwt                     2.0.1              pyhd8ed1ab_0    conda-forge/label/main
   pyopenssl                 20.0.1             pyhd3eb1b0_1
   pyparsing                 2.4.7              pyhd3eb1b0_0
   pyqt                      5.9.2            py38h05f1152_4
   pyrsistent                0.17.3           py38h7b6447c_0
   pysocks                   1.7.1            py38h06a4308_0
   python                    3.8.5                h7579374_1
   python-dateutil           2.8.1              pyhd3eb1b0_0
   python-editor             1.0.4                      py_0    conda-forge/label/main
   python_abi                3.8                      1_cp38    conda-forge/label/main
   pytz                      2020.5             pyhd3eb1b0_0
   pyyaml                    5.3.1            py38h8df0ef7_1    conda-forge/label/main
   pyzmq                     20.0.0           py38h2531618_1
   qt                        5.9.7                h5867ecd_1
   querystring_parser        1.2.4                      py_0    conda-forge/label/main
   readline                  8.1                  h27cfd23_0
   requests                  2.25.1             pyhd3eb1b0_0
   requests-oauthlib         1.3.0              pyh9f0ad1d_0    conda-forge/label/main
   rsa                       4.7                      pypi_0    pypi
   scipy                     1.5.2            py38h0b6359f_0
   seaborn                   0.11.1             pyhd3eb1b0_0
   send2trash                1.5.0              pyhd3eb1b0_1
   setuptools                52.0.0           py38h06a4308_0
   sip                       4.19.13          py38he6710b0_0
   six                       1.15.0           py38h06a4308_0
   smmap                     4.0.0              pyh44b312d_0    conda-forge/label/main
   snappy                    1.1.8                he6710b0_0
   sqlalchemy                1.3.20           py38h1e0a361_0    conda-forge/label/main
   sqlite                    3.33.0               h62c20be_0
   sqlparse                  0.4.1              pyh9f0ad1d_0    conda-forge/label/main
   tabulate                  0.8.7              pyh9f0ad1d_0    conda-forge/label/main
   tb-nightly                2.5.0a20210130           pypi_0    pypi
   tensorboard-plugin-wit    1.8.0                    pypi_0    pypi
   termcolor                 1.1.0                    pypi_0    pypi
   terminado                 0.9.2            py38h06a4308_0
   testpath                  0.4.4              pyhd3eb1b0_0
   tf-estimator-nightly      2.5.0.dev2021020101          pypi_0    pypi
   tf-nightly                2.5.0.dev20210130          pypi_0    pypi
   tifffile                  2021.1.14          pyhd3eb1b0_1
   tk                        8.6.10               hbc83047_0
   tornado                   6.1              py38h27cfd23_0
   traitlets                 5.0.5              pyhd3eb1b0_0
   typing-extensions         3.7.4.3                  pypi_0    pypi
   uncertainties             3.1.5              pyhd8ed1ab_0    conda-forge/label/main
   urllib3                   1.26.3             pyhd3eb1b0_0
   wcwidth                   0.2.5                      py_0
   webencodings              0.5.1                    py38_1
   websocket-client          0.57.0           py38h578d9bd_4    conda-forge/label/main
   werkzeug                  1.0.1              pyh9f0ad1d_0    conda-forge/label/main
   wheel                     0.36.2             pyhd3eb1b0_0
   wrapt                     1.12.1                   pypi_0    pypi
   xz                        5.2.5                h7b6447c_0
   yaml                      0.2.5                h516909a_0    conda-forge/label/main
   zeromq                    4.3.3                he6710b0_3
   zfp                       0.5.5                h2531618_4
   zipp                      3.4.0              pyhd3eb1b0_0
   zlib                      1.2.11               h7b6447c_3
   zstd                      1.4.5                h9ceee32_0
   
   Note: you may need to restart the kernel to use updated packages.
   {'SLURM_CHECKPOINT_IMAGE_DIR': '/var/slurm/checkpoint',
    'SLURM_NODELIST': 'node260',
    'SLURM_JOB_NAME': 'bash',
    'XDG_SESSION_ID': '595',
    'SLURMD_NODENAME': 'node260',
    'SLURM_TOPOLOGY_ADDR': 'node260',
    'SLURM_NTASKS_PER_NODE': '24',
    'HOSTNAME': 'login01',
    'SLURM_PRIO_PROCESS': '0',
    'SLURM_SRUN_COMM_PORT': '39229',
    'SHELL': '/bin/bash',
    'TERM': 'xterm-color',
    'SLURM_JOB_QOS': 'qstand',
    'SLURM_PTY_WIN_ROW': '39',
    'HISTSIZE': '1000',
    'TMPDIR': '/tmp',
    'SLURM_TOPOLOGY_ADDR_PATTERN': 'node',
    'SSH_CLIENT': '10.231.182.213 44428 22',
    'CONDA_SHLVL': '2',
    'CONDA_PROMPT_MODIFIER': '(tf-nightly) ',
    'QTDIR': '/usr/lib64/qt-3.3',
    'QTINC': '/usr/lib64/qt-3.3/include',
    'SSH_TTY': '/dev/pts/2',
    'NO_PROXY': 'localhost,127.0.0.0/8,.uni-jena.de,141.35.0.0/16,10.0.0.0/8,192.168.0.0/16,172.0.0.0/8,fe80::/7,2001:638:1558::/24',
    'QT_GRAPHICSSYSTEM_CHECKED': '1',
    'SLURM_NNODES': '1',
    'USER': 'ye53nis',
    'http_proxy': 'http://internet4nzm.rz.uni-jena.de:3128',
    'LS_COLORS': 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:',
    'CONDA_EXE': '/cluster/miniconda3/bin/conda',
    'SLURM_STEP_NUM_NODES': '1',
    'SLURM_JOBID': '652176',
    'SRUN_DEBUG': '3',
    'FTP_PROXY': 'http://internet4nzm.rz.uni-jena.de:3128',
    'ftp_proxy': 'http://internet4nzm.rz.uni-jena.de:3128',
    'SLURM_NTASKS': '24',
    'SLURM_LAUNCH_NODE_IPADDR': '192.168.192.5',
    'SLURM_STEP_ID': '0',
    'TMUX': '/tmp/tmux-67339/default,43792,2',
    '_CE_CONDA': '',
    'CONDA_PREFIX_1': '/cluster/miniconda3',
    'SLURM_STEP_LAUNCHER_PORT': '39229','SLURM_TASKS_PER_NODE': '24',
   'MAIL': '/var/spool/mail/ye53nis',
   'PATH': '/home/ye53nis/.conda/envs/tf-nightly/bin:/home/lex/Programme/miniconda3/envs/tf-nightly-lab/bin:/home/lex/Programme/miniconda3/condabin:/home/lex/.local/bin:/bin:/usr/bin:/usr/local/bin:/usr/local/sbin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/var/lib/snapd/snap/bin:/usr/sbin:/home/ye53nis/.local/bin:/home/ye53nis/bin',
   'SLURM_WORKING_CLUSTER': 'hpc:192.168.192.1:6817:8448',
   'SLURM_JOB_ID': '652176',
   'CONDA_PREFIX': '/home/ye53nis/.conda/envs/tf-nightly',
   'SLURM_JOB_USER': 'ye53nis',
   'SLURM_STEPID': '0',
   'PWD': '/',
   'SLURM_SRUN_COMM_HOST': '192.168.192.5',
   'LANG': 'en_US.UTF-8',
   'SLURM_PTY_WIN_COL': '205',
   'SLURM_UMASK': '0022',
   'MODULEPATH': '/usr/share/Modules/modulefiles:/etc/modulefiles:/cluster/modulefiles',
   'SLURM_JOB_UID': '67339',
   'LOADEDMODULES': '',
   'SLURM_NODEID': '0',
   'TMUX_PANE': '%2',
   'SLURM_SUBMIT_DIR': '/',
   'SLURM_TASK_PID': '167340',
   'SLURM_NPROCS': '24',
   'SLURM_CPUS_ON_NODE': '24',
   'SLURM_DISTRIBUTION': 'block',
   'HTTPS_PROXY': 'http://internet4nzm.rz.uni-jena.de:3128',
   'https_proxy': 'http://internet4nzm.rz.uni-jena.de:3128',
   'SLURM_PROCID': '0',
   'HISTCONTROL': 'ignoredups',
   '_CE_M': '',
   'SLURM_JOB_NODELIST': 'node260',
   'SLURM_PTY_PORT': '33620',
   'HOME': '/home/ye53nis',
   'SHLVL': '3',
   'SLURM_LOCALID': '0',
   'SLURM_JOB_GID': '13280',
   'SLURM_JOB_CPUS_PER_NODE': '24',
   'SLURM_CLUSTER_NAME': 'hpc',
   'no_proxy': 'localhost,127.0.0.0/8,.uni-jena.de,141.35.0.0/16,10.0.0.0/8,192.168.0.0/16,172.0.0.0/8,fe80::/7,2001:638:1558::/24',
   'SLURM_GTIDS': '0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23',
   'SLURM_SUBMIT_HOST': 'login01',
   'HTTP_PROXY': 'http://internet4nzm.rz.uni-jena.de:3128',
   'SLURM_JOB_PARTITION': 's_standard',
   'MATHEMATICA_HOME': '/cluster/apps/mathematica/11.3',
   'CONDA_PYTHON_EXE': '/cluster/miniconda3/bin/python',
   'LOGNAME': 'ye53nis',
   'SLURM_STEP_NUM_TASKS': '24',
   'QTLIB': '/usr/lib64/qt-3.3/lib',
   'SLURM_JOB_ACCOUNT': 'iaob',
   'SLURM_JOB_NUM_NODES': '1',
   'MODULESHOME': '/usr/share/Modules',
   'CONDA_DEFAULT_ENV': 'tf-nightly',
   'LESSOPEN': '||/usr/bin/lesspipe.sh %s',
   'SLURM_STEP_TASKS_PER_NODE': '24',
   'PORT': '9999',
   'SLURM_STEP_NODELIST': 'node260',
   'DISPLAY': ':0',
   'XDG_RUNTIME_DIR': '',
   'XAUTHORITY': '/home/lex/.Xauthority',
   'BASH_FUNC_module()': '() {  eval `/usr/bin/modulecmd bash $*`\n}',
   '_': '/home/ye53nis/.conda/envs/tf-nightly/bin/jupyter',
   'JPY_PARENT_PID': '446797',
   'CLICOLOR': '1',
   'PAGER': 'cat',
   'GIT_PAGER': 'cat',
   'MPLBACKEND': 'module://ipykernel.pylab.backend_inline'}
   

• mlflow environment variables

  conda activate tf-nightly
  cd /beegfs/ye53nis/drmed-git
  export MLFLOW_EXPERIMENT_NAME=exp-210204-unet
  export MLFLOW_TRACKING_URI=file:./data/mlruns
  mkdir data/exp-210204-unet
  

  (tf-nightly) [ye53nis@node117 drmed-git]$
  

• Use whole dataset (6400 training, 1600 validation, 2000 test), but during training, use only 1/5th of it per epoch - but this time with more balanced test set.

  mlflow run . -P fluotracify_path=/beegfs/ye53nis/drmed-git/src/ -P epochs=50 -P learning_rate=None -P csv_path_train=/beegfs/ye53nis/saves/firstartifact_Nov2020_train -P csv_path_test=/beegfs/ye53nis/saves/firstartifact_Nov2020_test -P steps_per_epoch=1280 -P validation_steps=320
  

  (tf-nightly) [ye53nis@node117 drmed-git]$ mlflow run . -P fluotracify_path=/beegfs/ye53nis/drmed-git/src/ -P epochs=50 -P learning_rate=None -P csv_path_train=/beegfs/ye53nis/saves/firstartifact_Nov2020_train -P csv_path_test=/beegfs/y
  e53nis/saves/firstartifact_Nov2020_test -P steps_per_epoch=1280 -P validation_steps=320
  WARNING:root:Malformed experiment '1'. Detailed error Yaml file './data/mlruns/1/meta.yaml' does not exist.
  Traceback (most recent call last):
    File "/home/ye53nis/.conda/envs/tf-nightly/lib/python3.8/site-packages/mlflow/store/tracking/file_store.py", line 237, in list_experiments
      experiment = self._get_experiment(exp_id, view_type)
    File "/home/ye53nis/.conda/envs/tf-nightly/lib/python3.8/site-packages/mlflow/store/tracking/file_store.py", line 311, in _get_experiment
      meta = read_yaml(experiment_dir, FileStore.META_DATA_FILE_NAME)
    File "/home/ye53nis/.conda/envs/tf-nightly/lib/python3.8/site-packages/mlflow/utils/file_utils.py", line 170, in read_yaml
      raise MissingConfigException("Yaml file '%s' does not exist." % file_path)
  mlflow.exceptions.MissingConfigException: Yaml file './data/mlruns/1/meta.yaml' does not exist.
  WARNING:root:Malformed experiment '0'. Detailed error Yaml file './data/mlruns/0/meta.yaml' does not exist.
  Traceback (most recent call last):
    File "/home/ye53nis/.conda/envs/tf-nightly/lib/python3.8/site-packages/mlflow/store/tracking/file_store.py", line 237, in list_experiments
      experiment = self._get_experiment(exp_id, view_type)
    File "/home/ye53nis/.conda/envs/tf-nightly/lib/python3.8/site-packages/mlflow/store/tracking/file_store.py", line 311, in _get_experiment
      meta = read_yaml(experiment_dir, FileStore.META_DATA_FILE_NAME)
    File "/home/ye53nis/.conda/envs/tf-nightly/lib/python3.8/site-packages/mlflow/utils/file_utils.py", line 170, in read_yaml
      raise MissingConfigException("Yaml file '%s' does not exist." % file_path)
  mlflow.exceptions.MissingConfigException: Yaml file './data/mlruns/0/meta.yaml' does not exist.
  2021/03/16 22:26:14 INFO mlflow.projects.utils: === Created directory /tmp/tmp4e89kbom for downloading remote URIs passed to arguments of type 'path' ===
  2021/03/16 22:26:14 INFO mlflow.projects.backend.local: === Running command 'source /cluster/miniconda3/bin/../etc/profile.d/conda.sh && conda activate mlflow-fd9a200e5a24d4c79a0ff13be73ccb5141ed072c 1>&2 && python src/fluotracify/trai
  ning/train.py /beegfs/ye53nis/drmed-git/src 5 0.2 16384 None 50 /beegfs/ye53nis/saves/firstartifact_Nov2020_train /beegfs/ye53nis/saves/firstartifact_Nov2020_test 3 1280 320' in run with ID '3cec3f26ed2d4004978c4ec37c00fba0' ===
  2021-03-16 22:26:31.835407: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
  2021-03-16 22:26:31.835484: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
  2.5.0-dev20210130
  2021-03-16 22:27:03.495143: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
  2021-03-16 22:27:03.495217: W tensorflow/stream_executor/cuda/cuda_driver.cc:326] failed call to cuInit: UNKNOWN ERROR (303)
  2021-03-16 22:27:03.495359: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node117): /proc/driver/nvidia/version does not exist
  GPUs:  []
  train 0 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/50/traces_brightclust_Nov2020_D50_set005.csv
  train 1 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.6/traces_brightclust_Nov2020_D0.6_set001.csv
  train 2 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/3.0/traces_brightclust_Nov2020_D3.0_set005.csv
  train 3 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/50/traces_brightclust_Nov2020_D50_set008.csv
  train 4 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/10/traces_brightclust_Nov2020_D10_set004.csv
  train 5 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/1.0/traces_brightclust_Nov2020_D1.0_set004.csv
  train 6 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.08/traces_brightclust_Nov2020_D0.08_set004.csv
  train 7 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/50/traces_brightclust_Nov2020_D50_set004.csv
  train 8 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/50/traces_brightclust_Nov2020_D50_set009.csv
  train 9 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/1.0/traces_brightclust_Nov2020_D1.0_set001.csv
  train 10 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.4/traces_brightclust_Nov2020_D0.4_set002.csv
  train 11 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.4/traces_brightclust_Nov2020_D0.4_set006.csv
  train 12 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.4/traces_brightclust_Nov2020_D0.4_set010.csv
  train 13 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.4/traces_brightclust_Nov2020_D0.4_set007.csv
  train 14 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.2/traces_brightclust_Nov2020_D0.2_set004.csv
  train 15 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/10/traces_brightclust_Nov2020_D10_set008.csv
  train 16 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/3.0/traces_brightclust_Nov2020_D3.0_set009.csv
  train 17 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/1.0/traces_brightclust_Nov2020_D1.0_set002.csv
  train 18 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.069/traces_brightclust_Nov2020_D0.069_set008.csv
  train 19 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.2/traces_brightclust_Nov2020_D0.2_set010.csv
  train 20 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.4/traces_brightclust_Nov2020_D0.4_set003.csv
  train 21 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.069/traces_brightclust_Nov2020_D0.069_set006.csv
  train 22 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/10/traces_brightclust_Nov2020_D10_set010.csv
  train 23 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.08/traces_brightclust_Nov2020_D0.08_set009.csv
  train 24 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.069/traces_brightclust_Nov2020_D0.069_set007.csv
  train 25 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/10/traces_brightclust_Nov2020_D10_set009.csv
  train 26 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/50/traces_brightclust_Nov2020_D50_set010.csv
  train 27 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.08/traces_brightclust_Nov2020_D0.08_set008.csv
  train 28 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.6/traces_brightclust_Nov2020_D0.6_set010.csv
  train 29 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/10/traces_brightclust_Nov2020_D10_set003.csv
  train 30 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/1.0/traces_brightclust_Nov2020_D1.0_set008.csv
  train 31 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/3.0/traces_brightclust_Nov2020_D3.0_set003.csv
  train 32 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.4/traces_brightclust_Nov2020_D0.4_set009.csv
  train 33 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.1/traces_brightclust_Nov2020_D0.1_set004.csv
  train 34 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/3.0/traces_brightclust_Nov2020_D3.0_set001.csv
  train 35 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.1/traces_brightclust_Nov2020_D0.1_set010.csv
  train 36 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.4/traces_brightclust_Nov2020_D0.4_set004.csv
  train 37 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/10/traces_brightclust_Nov2020_D10_set007.csv
  train 38 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.6/traces_brightclust_Nov2020_D0.6_set002.csv
  train 39 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.1/traces_brightclust_Nov2020_D0.1_set009.csv
  train 40 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.069/traces_brightclust_Nov2020_D0.069_set002.csv
  train 41 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.1/traces_brightclust_Nov2020_D0.1_set006.csv
  train 42 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.069/traces_brightclust_Nov2020_D0.069_set009.csv
  train 43 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.1/traces_brightclust_Nov2020_D0.1_set007.csv
  train 44 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.2/traces_brightclust_Nov2020_D0.2_set001.csv
  train 45 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/50/traces_brightclust_Nov2020_D50_set007.csv
  train 46 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.08/traces_brightclust_Nov2020_D0.08_set006.csv
  train 47 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.1/traces_brightclust_Nov2020_D0.1_set003.csv
  train 48 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.2/traces_brightclust_Nov2020_D0.2_set009.csv
  train 49 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.6/traces_brightclust_Nov2020_D0.6_set005.csv
  train 50 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.1/traces_brightclust_Nov2020_D0.1_set008.csv
  train 51 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/3.0/traces_brightclust_Nov2020_D3.0_set006.csv
  train 52 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/1.0/traces_brightclust_Nov2020_D1.0_set009.csv
  train 53 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/3.0/traces_brightclust_Nov2020_D3.0_set010.csv
  train 54 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.6/traces_brightclust_Nov2020_D0.6_set007.csv
  train 55 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.6/traces_brightclust_Nov2020_D0.6_set006.csv
  train 56 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.069/traces_brightclust_Nov2020_D0.069_set003.csv
  train 57 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.08/traces_brightclust_Nov2020_D0.08_set010.csv
  train 58 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/3.0/traces_brightclust_Nov2020_D3.0_set008.csv
  train 59 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.2/traces_brightclust_Nov2020_D0.2_set008.csv
  train 60 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.08/traces_brightclust_Nov2020_D0.08_set002.csv
  train 61 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.2/traces_brightclust_Nov2020_D0.2_set006.csv
  train 62 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.6/traces_brightclust_Nov2020_D0.6_set004.csv
  train 63 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/10/traces_brightclust_Nov2020_D10_set006.csv
  train 64 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.2/traces_brightclust_Nov2020_D0.2_set003.csv
  train 65 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.08/traces_brightclust_Nov2020_D0.08_set007.csv
  train 66 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/50/traces_brightclust_Nov2020_D50_set006.csv
  train 67 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/1.0/traces_brightclust_Nov2020_D1.0_set010.csv
  train 68 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/1.0/traces_brightclust_Nov2020_D1.0_set007.csv
  train 0 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.069_set010.csv
  train 1 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D50_set002.csv
  train 2 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.4_set005.csv
  train 3 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.2_set005.csv
  train 4 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D3.0_set007.csv
  train 5 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D3.0_set002.csv
  train 6 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D50_set001.csv
  train 7 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.2_set007.csv
  train 8 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.6_set009.csv
  train 9 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D10_set002.csv
  train 10 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.08_set005.csv
  train 11 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.6_set008.csv
  train 12 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.1_set005.csv
  train 13 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.4_set008.csv
  train 14 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D10_set005.csv
  train 15 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D1.0_set006.csv
  train 16 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.069_set005.csv
  train 17 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D50_set003.csv
  train 18 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.1_set001.csv
  train 19 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.08_set003.csv
  train 20 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D1.0_set003.csv
  train 21 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D1.0_set005.csv
  train 22 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.2_set002.csv
  train 23 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.1_set002.csv
  train 24 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D3.0_set004.csv
  train 25 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.08_set001.csv
  train 26 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.069_set001.csv
  train 27 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D10_set001.csv
  train 28 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.6_set003.csv
  train 29 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.4_set001.csv
  The given DataFrame was split into 3 parts with shapes: [(16384, 6900), (16384, 6900), (16384, 6900)]
  The given DataFrame was split into 3 parts with shapes: [(16384, 3000), (16384, 3000), (16384, 3000)]
  
  for each 16384 timestap trace there are the following numbers of corrupted timesteps:
  label001_1    1916
  label002_1    1004
  label003_1    1476
  label004_1    1154
  label005_1    1454
  dtype: int64
  2021-03-16 22:34:23.544689: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operati
  ons:  AVX2 FMA
  To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
  number of training examples: 5520, number of validation examples: 1380
  
  ------------------------
  number of test examples: 3000
  
  input - shape:   (None, 16384, 1)
  output - shape:  (None, 16384, 1)
  2021-03-16 22:34:30.005739: I tensorflow/core/profiler/lib/profiler_session.cc:126] Profiler session initializing.
  2021-03-16 22:34:30.005803: I tensorflow/core/profiler/lib/profiler_session.cc:141] Profiler session started.
  2021-03-16 22:34:30.005880: I tensorflow/core/profiler/lib/profiler_session.cc:158] Profiler session tear down.
  2021/03/16 22:34:30 INFO mlflow.utils.autologging_utils: tensorflow autologging will track hyperparameters, performance metrics, model artifacts, and lineage information for the current tensorflow workflow to the MLflow run with ID '3c
  ec3f26ed2d4004978c4ec37c00fba0'
  2021/03/16 22:34:30 WARNING mlflow.utils.autologging_utils: MLflow issued a warning during tensorflow autologging: "/home/ye53nis/.conda/envs/mlflow-fd9a200e5a24d4c79a0ff13be73ccb5141ed072c/lib/python3.8/site-packages/mlflow/utils/auto
  logging_utils.py:86: UserWarning: Logging to MLflow failed: Changing param values is not allowed. Param with key='batch_size' was already logged with value='5' for run ID='3cec3f26ed2d4004978c4ec37c00fba0'. Attempted logging new value
  'None'."
  Epoch 1/50
  2021-03-16 22:34:43.027454: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:145] None of the MLIR Optimization Passes are enabled (registered 2)
  2021-03-16 22:34:43.259103: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 2194880000 Hz
     1/1280 [..............................] - ETA: 7:06:43 - loss: 1.7012 - tp0.1: 11443.0000 - fp0.1: 69168.0000 - tn0.1: 50.0000 - fn0.1: 1259.0000 - precision0.1: 0.1420 - recall0.1: 0.9009 - tp0.3: 9371.0000 - fp0.3: 67996.0000 - tn
  0.3: 1222.0000 - fn0.3: 3331.0000 - precision0.3: 0.1211 - recall0.3: 0.7378 - tp0.5: 5565.0000 - fp0.5: 37117.0000 - tn0.5: 32101.0000 - fn0.5: 7137.0000 - precision0.5: 0.1304 - recall0.5: 0.4381 - tp0.7: 1504.0000 - fp0.7: 2293.0000
   - tn0.7: 66925.0000 - fn0.7: 11198.0000 - precision0.7: 0.3961 - recall0.7: 0.1184 - tp0.9: 284.0000 - fp0.9: 114.0000 - tn0.9: 69104.0000 - fn0.9: 12418.0000 - precision0.9: 0.7136 - recall0.9: 0.0224 - accuracy: 0.4598 - auc: 0.4176
  2021-03-16 22:34:50.791836: I tensorflow/core/profiler/lib/profiler_session.cc:126] Profiler session initializing.
  2021-03-16 22:34:50.791935: I tensorflow/core/profiler/lib/profiler_session.cc:141] Profiler session started.
     2/1280 [..............................] - ETA: 55:09 - loss: 1.5706 - tp0.1: 15933.5000 - fp0.1: 91410.0000 - tn0.1: 14176.5000 - fn0.1: 1360.0000 - precision0.1: 0.1471 - recall0.1: 0.9171 - tp0.3: 11355.0000 - fp0.3: 68009.0000 -
  tn0.3: 37577.5000 - fn0.3: 5938.5000 - precision0.3: 0.1425 - recall0.3: 0.6736 - tp0.5: 7264.0000 - fp0.5: 37126.5000 - tn0.5: 68460.0000 - fn0.5: 10029.5000 - precision0.5: 0.1624 - recall0.5: 0.4238 - tp0.7: 2845.0000 - fp0.7: 2300.
  0000 - tn0.7: 103286.5000 - fn0.7: 14448.5000 - precision0.7: 0.5204 - recall0.7: 0.1548 - tp0.9: 1306.5000 - fp0.9: 119.0000 - tn0.9: 105467.5000 - fn0.9: 15987.0000 - precision0.9: 0.8315 - recall0.9: 0.0644 - accuracy: 0.5771 - auc:
   0.53722021-03-16 22:34:53.384001: I tensorflow/core/profiler/lib/profiler_session.cc:66] Profiler session collecting data.
  2021-03-16 22:34:53.414299: I tensorflow/core/profiler/lib/profiler_session.cc:158] Profiler session tear down.
  2021-03-16 22:34:53.449110: I tensorflow/core/profiler/rpc/client/save_profile.cc:137] Creating directory: /tmp/tb/train/plugins/profile/2021_03_16_22_34_53
  2021-03-16 22:34:53.468888: I tensorflow/core/profiler/rpc/client/save_profile.cc:143] Dumped gzipped tool data for trace.json.gz to /tmp/tb/train/plugins/profile/2021_03_16_22_34_53/node117.trace.json.gz
  2021-03-16 22:34:53.498618: I tensorflow/core/profiler/rpc/client/save_profile.cc:137] Creating directory: /tmp/tb/train/plugins/profile/2021_03_16_22_34_53
  2021-03-16 22:34:53.498775: I tensorflow/core/profiler/rpc/client/save_profile.cc:143] Dumped gzipped tool data for memory_profile.json.gz to /tmp/tb/train/plugins/profile/2021_03_16_22_34_53/node117.memory_profile.json.gz
  2021-03-16 22:34:53.501218: I tensorflow/core/profiler/rpc/client/capture_profile.cc:251] Creating directory: /tmp/tb/train/plugins/profile/2021_03_16_22_34_53Dumped tool data for xplane.pb to /tmp/tb/train/plugins/profile/2021_03_16_2
  2_34_53/node117.xplane.pb
  Dumped tool data for overview_page.pb to /tmp/tb/train/plugins/profile/2021_03_16_22_34_53/node117.overview_page.pb
  Dumped tool data for input_pipeline.pb to /tmp/tb/train/plugins/profile/2021_03_16_22_34_53/node117.input_pipeline.pb
  Dumped tool data for tensorflow_stats.pb to /tmp/tb/train/plugins/profile/2021_03_16_22_34_53/node117.tensorflow_stats.pb
  Dumped tool data for kernel_stats.pb to /tmp/tb/train/plugins/profile/2021_03_16_22_34_53/node117.kernel_stats.pb
  
  1280/1280 [==============================] - 3356s 3s/step - loss: 0.9438 - tp0.1: 7090847.5004 - fp0.1: 12587220.8876 - tn0.1: 31719449.0960 - fn0.1: 1113141.3841 - precision0.1: 0.3522 - recall0.1: 0.8539 - tp0.3: 5944647.5215 - fp0.
  3: 7704502.4426 - tn0.3: 36602146.0117 - fn0.3: 2259341.3630 - precision0.3: 0.4254 - recall0.3: 0.6980 - tp0.5: 3390349.7166 - fp0.5: 2172177.0687 - tn0.5: 42134502.0273 - fn0.5: 4813639.1678 - precision0.5: 0.5845 - recall0.5: 0.4054
   - tp0.7: 2075799.0164 - fp0.7: 570841.5262 - tn0.7: 43735856.3763 - fn0.7: 6128189.8681 - precision0.7: 0.7474 - recall0.7: 0.2354 - tp0.9: 1317276.8236 - fp0.9: 157422.0055 - tn0.9: 44149241.1343 - fn0.9: 6886712.0609 - precision0.9:
   0.8734 - recall0.9: 0.1441 - accuracy: 0.8611 - auc: 0.8419 - val_loss: 1.7438 - val_tp0.1: 1858818.0000 - val_fp0.1: 1751074.0000 - val_tn0.1: 20297196.0000 - val_fn0.1: 2307317.0000 - val_precision0.1: 0.5149 - val_recall0.1: 0.4462
   - val_tp0.3: 1147351.0000 - val_fp0.3: 857162.0000 - val_tn0.3: 21191096.0000 - val_fn0.3: 3018784.0000 - val_precision0.3: 0.5724 - val_recall0.3: 0.2754 - val_tp0.5: 22832.0000 - val_fp0.5: 2666.0000 - val_tn0.5: 22045600.0000 - val
  _fn0.5: 4143303.0000 - val_precision0.5: 0.8954 - val_recall0.5: 0.0055 - val_tp0.7: 0.0000e+00 - val_fp0.7: 0.0000e+00 - val_tn0.7: 22048264.0000 - val_fn0.7: 4166135.0000 - val_precision0.7: 0.0000e+00 - val_recall0.7: 0.0000e+00 - v
  al_tp0.9: 0.0000e+00 - val_fp0.9: 0.0000e+00 - val_tn0.9: 22048264.0000 - val_fn0.9: 4166135.0000 - val_precision0.9: 0.0000e+00 - val_recall0.9: 0.0000e+00 - val_accuracy: 0.8418 - val_auc: 0.7129
  Epoch 2/50
  1280/1280 [==============================] - 3273s 3s/step - loss: 0.7454 - tp0.1: 7319058.9383 - fp0.1: 12182101.0671 - tn0.1: 32209636.0234 - fn0.1: 799871.6300 - precision0.1: 0.3732 - recall0.1: 0.9003 - tp0.3: 6349950.7744 - fp0.3
  : 7864965.2303 - tn0.3: 36526747.8587 - fn0.3: 1768979.7939 - precision0.3: 0.4426 - recall0.3: 0.7794 - tp0.5: 3304850.4598 - fp0.5: 1018320.5308 - tn0.5: 43373388.4637 - fn0.5: 4814080.1085 - precision0.5: 0.7573 - recall0.5: 0.4070
  - tp0.7: 2713284.5909 - fp0.7: 370786.4668 - tn0.7: 44020933.5667 - fn0.7: 5405645.9774 - precision0.7: 0.8821 - recall0.7: 0.3298 - tp0.9: 2048963.3240 - fp0.9: 143660.5839 - tn0.9: 44248083.9859 - fn0.9: 6069967.2443 - precision0.9:
  0.9325 - recall0.9: 0.2463 - accuracy: 0.8888 - auc: 0.8781 - val_loss: 1.8629 - val_tp0.1: 1696755.0000 - val_fp0.1: 1369278.0000 - val_tn0.1: 20627520.0000 - val_fn0.1: 2520850.0000 - val_precision0.1: 0.5534 - val_recall0.1: 0.4023
  - val_tp0.3: 1177823.0000 - val_fp0.3: 724329.0000 - val_tn0.3: 21272468.0000 - val_fn0.3: 3039782.0000 - val_precision0.3: 0.6192 - val_recall0.3: 0.2793 - val_tp0.5: 726105.0000 - val_fp0.5: 220632.0000 - val_tn0.5: 21776148.0000 - v
  al_fn0.5: 3491500.0000 - val_precision0.5: 0.7670 - val_recall0.5: 0.1722 - val_tp0.7: 622624.0000 - val_fp0.7: 149383.0000 - val_tn0.7: 21847414.0000 - val_fn0.7: 3594981.0000 - val_precision0.7: 0.8065 - val_recall0.7: 0.1476 - val_t
  p0.9: 478068.0000 - val_fp0.9: 78711.0000 - val_tn0.9: 21918088.0000 - val_fn0.9: 3739537.0000 - val_precision0.9: 0.8586 - val_recall0.9: 0.1134 - val_accuracy: 0.8584 - val_auc: 0.6881
  Epoch 3/50
  1280/1280 [==============================] - 3268s 3s/step - loss: 0.6401 - tp0.1: 7574454.9469 - fp0.1: 10689439.2311 - tn0.1: 33573907.5972 - fn0.1: 672847.0898 - precision0.1: 0.4156 - recall0.1: 0.9171 - tp0.3: 6542181.5066 - fp0.3
  : 5029903.7557 - tn0.3: 39233452.5995 - fn0.3: 1705120.5301 - precision0.3: 0.5571 - recall0.3: 0.7912 - tp0.5: 4316700.4192 - fp0.5: 813585.7369 - tn0.5: 43449769.7229 - fn0.5: 3930601.6175 - precision0.5: 0.8494 - recall0.5: 0.5079 -
   tp0.7: 3862233.1499 - fp0.7: 434886.9867 - tn0.7: 43828435.6581 - fn0.7: 4385068.8868 - precision0.7: 0.9012 - recall0.7: 0.4570 - tp0.9: 3000635.4949 - fp0.9: 146691.9102 - tn0.9: 44116668.4348 - fn0.9: 5246666.5418 - precision0.9: 0
  .9538 - recall0.9: 0.3597 - accuracy: 0.9073 - auc: 0.9113 - val_loss: 1.2681 - val_tp0.1: 2112447.0000 - val_fp0.1: 1372411.0000 - val_tn0.1: 20688214.0000 - val_fn0.1: 2041318.0000 - val_precision0.1: 0.6062 - val_recall0.1: 0.5086 -
   val_tp0.3: 1459147.0000 - val_fp0.3: 526021.0000 - val_tn0.3: 21534608.0000 - val_fn0.3: 2694618.0000 - val_precision0.3: 0.7350 - val_recall0.3: 0.3513 - val_tp0.5: 1206684.0000 - val_fp0.5: 343698.0000 - val_tn0.5: 21716924.0000 - v
  al_fn0.5: 2947081.0000 - val_precision0.5: 0.7783 - val_recall0.5: 0.2905 - val_tp0.7: 926901.0000 - val_fp0.7: 212441.0000 - val_tn0.7: 21848180.0000 - val_fn0.7: 3226864.0000 - val_precision0.7: 0.8135 - val_recall0.7: 0.2231 - val_t
  p0.9: 701556.0000 - val_fp0.9: 119958.0000 - val_tn0.9: 21940680.0000 - val_fn0.9: 3452209.0000 - val_precision0.9: 0.8540 - val_recall0.9: 0.1689 - val_accuracy: 0.8745 - val_auc: 0.7587
  Epoch 4/50
  1280/1280 [==============================] - 3266s 3s/step - loss: 0.5483 - tp0.1: 7577427.0859 - fp0.1: 8351943.7939 - tn0.1: 35940757.2381 - fn0.1: 640533.9243 - precision0.1: 0.4704 - recall0.1: 0.9241 - tp0.3: 6848424.7369 - fp0.3:
   4027475.1538 - tn0.3: 40265212.3185 - fn0.3: 1369536.2732 - precision0.3: 0.6241 - recall0.3: 0.8352 - tp0.5: 6060996.3591 - fp0.5: 2313256.1897 - tn0.5: 41979441.1702 - fn0.5: 2156964.6511 - precision0.5: 0.7177 - recall0.5: 0.7389 -
   tp0.7: 4556251.4965 - fp0.7: 542255.6714 - tn0.7: 43750437.9961 - fn0.7: 3661709.5137 - precision0.7: 0.8912 - recall0.7: 0.5511 - tp0.9: 3395492.4715 - fp0.9: 154251.2139 - tn0.9: 44138439.1054 - fn0.9: 4822468.5386 - precision0.9: 0
  .9548 - recall0.9: 0.4106 - accuracy: 0.9134 - auc: 0.9349 - val_loss: 1.5831 - val_tp0.1: 3220872.0000 - val_fp0.1: 3814830.0000 - val_tn0.1: 18497596.0000 - val_fn0.1: 681105.0000 - val_precision0.1: 0.4578 - val_recall0.1: 0.8254 -
  val_tp0.3: 2365990.0000 - val_fp0.3: 2561983.0000 - val_tn0.3: 19750440.0000 - val_fn0.3: 1535987.0000 - val_precision0.3: 0.4801 - val_recall0.3: 0.6064 - val_tp0.5: 1952532.0000 - val_fp0.5: 2291714.0000 - val_tn0.5: 20020708.0000 -
  val_fn0.5: 1949445.0000 - val_precision0.5: 0.4600 - val_recall0.5: 0.5004 - val_tp0.7: 1618270.0000 - val_fp0.7: 2113505.0000 - val_tn0.7: 20198920.0000 - val_fn0.7: 2283707.0000 - val_precision0.7: 0.4336 - val_recall0.7: 0.4147 - va
  l_tp0.9: 1467404.0000 - val_fp0.9: 1996280.0000 - val_tn0.9: 20316148.0000 - val_fn0.9: 2434573.0000 - val_precision0.9: 0.4237 - val_recall0.9: 0.3761 - val_accuracy: 0.8382 - val_auc: 0.8531
  Epoch 5/50
  1280/1280 [==============================] - 3259s 3s/step - loss: 0.4575 - tp0.1: 7563591.0968 - fp0.1: 6166461.2155 - tn0.1: 38207695.0180 - fn0.1: 572899.0141 - precision0.1: 0.5449 - recall0.1: 0.9269 - tp0.3: 6976500.9703 - fp0.3:
   2853572.8470 - tn0.3: 41520584.7908 - fn0.3: 1159989.1405 - precision0.3: 0.7064 - recall0.3: 0.8492 - tp0.5: 6358931.4910 - fp0.5: 1545188.1085 - tn0.5: 42828968.9243 - fn0.5: 1777558.6198 - precision0.5: 0.8010 - recall0.5: 0.7718 -
   tp0.7: 5171541.8845 - fp0.7: 510104.5824 - tn0.7: 43864075.8119 - fn0.7: 2964948.2264 - precision0.7: 0.9101 - recall0.7: 0.6254 - tp0.9: 3989419.1991 - fp0.9: 146867.6534 - tn0.9: 44227286.8977 - fn0.9: 4147070.9118 - precision0.9: 0
  .9658 - recall0.9: 0.4744 - accuracy: 0.9360 - auc: 0.9496 - val_loss: 1.0113 - val_tp0.1: 2745883.0000 - val_fp0.1: 1806365.0000 - val_tn0.1: 20277030.0000 - val_fn0.1: 1385129.0000 - val_precision0.1: 0.6032 - val_recall0.1: 0.6647 -
   val_tp0.3: 1898747.0000 - val_fp0.3: 1004741.0000 - val_tn0.3: 21078656.0000 - val_fn0.3: 2232265.0000 - val_precision0.3: 0.6540 - val_recall0.3: 0.4596 - val_tp0.5: 1378572.0000 - val_fp0.5: 684339.0000 - val_tn0.5: 21399046.0000 -
  val_fn0.5: 2752440.0000 - val_precision0.5: 0.6683 - val_recall0.5: 0.3337 - val_tp0.7: 1058802.0000 - val_fp0.7: 422419.0000 - val_tn0.7: 21660976.0000 - val_fn0.7: 3072210.0000 - val_precision0.7: 0.7148 - val_recall0.7: 0.2563 - val
  _tp0.9: 633251.0000 - val_fp0.9: 145724.0000 - val_tn0.9: 21937676.0000 - val_fn0.9: 3497761.0000 - val_precision0.9: 0.8129 - val_recall0.9: 0.1533 - val_accuracy: 0.8689 - val_auc: 0.8229
  Epoch 6/50
  1280/1280 [==============================] - 3293s 3s/step - loss: 0.3999 - tp0.1: 7670968.6714 - fp0.1: 5321303.1358 - tn0.1: 38997617.0039 - fn0.1: 520751.9563 - precision0.1: 0.5842 - recall0.1: 0.9342 - tp0.3: 7188663.1733 - fp0.3:
   2542544.6495 - tn0.3: 41776414.2037 - fn0.3: 1003057.4543 - precision0.3: 0.7341 - recall0.3: 0.8729 - tp0.5: 6622211.5550 - fp0.5: 1333118.7088 - tn0.5: 42985778.1749 - fn0.5: 1569509.0726 - precision0.5: 0.8305 - recall0.5: 0.8017 -
   tp0.7: 5590775.7127 - fp0.7: 478800.2568 - tn0.7: 43840116.4879 - fn0.7: 2600944.9149 - precision0.7: 0.9198 - recall0.7: 0.6746 - tp0.9: 4639602.3575 - fp0.9: 154184.4598 - tn0.9: 44164751.2225 - fn0.9: 3552118.2701 - precision0.9: 0
  .9678 - recall0.9: 0.5580 - accuracy: 0.9436 - auc: 0.9557 - val_loss: 0.7256 - val_tp0.1: 3301232.0000 - val_fp0.1: 2777637.0000 - val_tn0.1: 19396900.0000 - val_fn0.1: 738630.0000 - val_precision0.1: 0.5431 - val_recall0.1: 0.8172 -
  val_tp0.3: 2789396.0000 - val_fp0.3: 1963405.0000 - val_tn0.3: 20211132.0000 - val_fn0.3: 1250466.0000 - val_precision0.3: 0.5869 - val_recall0.3: 0.6905 - val_tp0.5: 2332065.0000 - val_fp0.5: 1478069.0000 - val_tn0.5: 20696466.0000 -
  val_fn0.5: 1707797.0000 - val_precision0.5: 0.6121 - val_recall0.5: 0.5773 - val_tp0.7: 1949962.0000 - val_fp0.7: 1076978.0000 - val_tn0.7: 21097560.0000 - val_fn0.7: 2089900.0000 - val_precision0.7: 0.6442 - val_recall0.7: 0.4827 - va
  l_tp0.9: 1380432.0000 - val_fp0.9: 492826.0000 - val_tn0.9: 21681716.0000 - val_fn0.9: 2659430.0000 - val_precision0.9: 0.7369 - val_recall0.9: 0.3417 - val_accuracy: 0.8785 - val_auc: 0.8740
  Epoch 7/50
  1280/1280 [==============================] - 3286s 3s/step - loss: 0.3597 - tp0.1: 7890232.5785 - fp0.1: 4953542.9781 - tn0.1: 39218777.5909 - fn0.1: 448103.7330 - precision0.1: 0.6178 - recall0.1: 0.9465 - tp0.3: 7467308.2701 - fp0.3:
   2391275.9922 - tn0.3: 41781019.9024 - fn0.3: 871028.0414 - precision0.3: 0.7613 - recall0.3: 0.8946 - tp0.5: 6988139.4286 - fp0.5: 1302055.8244 - tn0.5: 42870220.7486 - fn0.5: 1350196.8829 - precision0.5: 0.8451 - recall0.5: 0.8377 -
  tp0.7: 5947552.7845 - fp0.7: 472146.9633 - tn0.7: 43700160.4223 - fn0.7: 2390783.5269 - precision0.7: 0.9259 - recall0.7: 0.7138 - tp0.9: 4968978.2139 - fp0.9: 143981.8579 - tn0.9: 44028364.7588 - fn0.9: 3369358.0976 - precision0.9: 0.
  9713 - recall0.9: 0.5953 - accuracy: 0.9493 - auc: 0.9636 - val_loss: 0.7770 - val_tp0.1: 3234726.0000 - val_fp0.1: 2249439.0000 - val_tn0.1: 19829152.0000 - val_fn0.1: 901082.0000 - val_precision0.1: 0.5898 - val_recall0.1: 0.7821 - v
  al_tp0.3: 2554897.0000 - val_fp0.3: 1502825.0000 - val_tn0.3: 20575758.0000 - val_fn0.3: 1580911.0000 - val_precision0.3: 0.6296 - val_recall0.3: 0.6178 - val_tp0.5: 1992390.0000 - val_fp0.5: 1136809.0000 - val_tn0.5: 20941788.0000 - v
  al_fn0.5: 2143418.0000 - val_precision0.5: 0.6367 - val_recall0.5: 0.4817 - val_tp0.7: 1526404.0000 - val_fp0.7: 689435.0000 - val_tn0.7: 21389156.0000 - val_fn0.7: 2609404.0000 - val_precision0.7: 0.6889 - val_recall0.7: 0.3691 - val_
  tp0.9: 847679.0000 - val_fp0.9: 133403.0000 - val_tn0.9: 21945180.0000 - val_fn0.9: 3288129.0000 - val_precision0.9: 0.8640 - val_recall0.9: 0.2050 - val_accuracy: 0.8749 - val_auc: 0.8627
  Epoch 8/50
  1280/1280 [==============================] - 3289s 3s/step - loss: 0.3339 - tp0.1: 7747141.5660 - fp0.1: 4489700.9196 - tn0.1: 39841447.7486 - fn0.1: 432355.7814 - precision0.1: 0.6332 - recall0.1: 0.9472 - tp0.3: 7357958.1397 - fp0.3:
   2241207.2966 - tn0.3: 42089942.4052 - fn0.3: 821539.2077 - precision0.3: 0.7676 - recall0.3: 0.9001 - tp0.5: 6904184.2022 - fp0.5: 1214222.3841 - tn0.5: 43116930.9828 - fn0.5: 1275313.1452 - precision0.5: 0.8526 - recall0.5: 0.8448 -
  tp0.7: 5918119.9930 - fp0.7: 436485.9844 - tn0.7: 43894677.5480 - fn0.7: 2261377.3544 - precision0.7: 0.9324 - recall0.7: 0.7243 - tp0.9: 4978028.0929 - fp0.9: 136413.7908 - tn0.9: 44194744.8298 - fn0.9: 3201469.2545 - precision0.9: 0.
  9737 - recall0.9: 0.6094 - accuracy: 0.9533 - auc: 0.9664 - val_loss: 0.7192 - val_tp0.1: 3266754.0000 - val_fp0.1: 2261077.0000 - val_tn0.1: 19887774.0000 - val_fn0.1: 798799.0000 - val_precision0.1: 0.5910 - val_recall0.1: 0.8035 - v
  al_tp0.3: 2594715.0000 - val_fp0.3: 1533904.0000 - val_tn0.3: 20614936.0000 - val_fn0.3: 1470838.0000 - val_precision0.3: 0.6285 - val_recall0.3: 0.6382 - val_tp0.5: 2022214.0000 - val_fp0.5: 1204526.0000 - val_tn0.5: 20944316.0000 - v
  al_fn0.5: 2043339.0000 - val_precision0.5: 0.6267 - val_recall0.5: 0.4974 - val_tp0.7: 1633266.0000 - val_fp0.7: 893751.0000 - val_tn0.7: 21255104.0000 - val_fn0.7: 2432287.0000 - val_precision0.7: 0.6463 - val_recall0.7: 0.4017 - val_
  tp0.9: 1072066.0000 - val_fp0.9: 366453.0000 - val_tn0.9: 21782404.0000 - val_fn0.9: 2993487.0000 - val_precision0.9: 0.7453 - val_recall0.9: 0.2637 - val_accuracy: 0.8761 - val_auc: 0.8749
  Epoch 9/50
  1280/1280 [==============================] - 3297s 3s/step - loss: 0.3316 - tp0.1: 7830543.1632 - fp0.1: 4449068.0055 - tn0.1: 39786384.2771 - fn0.1: 444679.2272 - precision0.1: 0.6389 - recall0.1: 0.9467 - tp0.3: 7430622.2623 - fp0.3:
   2239407.0976 - tn0.3: 41996012.4926 - fn0.3: 844600.1280 - precision0.3: 0.7686 - recall0.3: 0.8983 - tp0.5: 6975903.1639 - fp0.5: 1214978.6417 - tn0.5: 43020456.3084 - fn0.5: 1299319.2264 - precision0.5: 0.8518 - recall0.5: 0.8438 -
  tp0.7: 6015111.5628 - fp0.7: 439544.0406 - tn0.7: 43795920.7791 - fn0.7: 2260110.8275 - precision0.7: 0.9325 - recall0.7: 0.7281 - tp0.9: 5071827.3482 - fp0.9: 135401.1327 - tn0.9: 44100020.3169 - fn0.9: 3203395.0422 - precision0.9: 0.
  9746 - recall0.9: 0.6139 - accuracy: 0.9520 - auc: 0.9664 - val_loss: 0.7987 - val_tp0.1: 3338155.0000 - val_fp0.1: 2467457.0000 - val_tn0.1: 19568284.0000 - val_fn0.1: 840502.0000 - val_precision0.1: 0.5750 - val_recall0.1: 0.7989 - v
  al_tp0.3: 2683614.0000 - val_fp0.3: 1892638.0000 - val_tn0.3: 20143100.0000 - val_fn0.3: 1495043.0000 - val_precision0.3: 0.5864 - val_recall0.3: 0.6422 - val_tp0.5: 2192750.0000 - val_fp0.5: 1587985.0000 - val_tn0.5: 20447752.0000 - v
  al_fn0.5: 1985907.0000 - val_precision0.5: 0.5800 - val_recall0.5: 0.5247 - val_tp0.7: 1808320.0000 - val_fp0.7: 1284650.0000 - val_tn0.7: 20751092.0000 - val_fn0.7: 2370337.0000 - val_precision0.7: 0.5847 - val_recall0.7: 0.4328 - val
  _tp0.9: 1198897.0000 - val_fp0.9: 618524.0000 - val_tn0.9: 21417220.0000 - val_fn0.9: 2979760.0000 - val_precision0.9: 0.6597 - val_recall0.9: 0.2869 - val_accuracy: 0.8637 - val_auc: 0.8621
  Epoch 10/50
  1280/1280 [==============================] - 3281s 3s/step - loss: 0.3275 - tp0.1: 7792627.5020 - fp0.1: 4279949.3427 - tn0.1: 40001537.3630 - fn0.1: 436537.2279 - precision0.1: 0.6422 - recall0.1: 0.9472 - tp0.3: 7406705.6987 - fp0.3:
   2160243.7174 - tn0.3: 42121245.1132 - fn0.3: 822459.0312 - precision0.3: 0.7727 - recall0.3: 0.8998 - tp0.5: 6961682.1405 - fp0.5: 1207985.9984 - tn0.5: 43073509.1913 - fn0.5: 1267482.5894 - precision0.5: 0.8511 - recall0.5: 0.8457 -
  tp0.7: 5977782.7900 - fp0.7: 441395.7190 - tn0.7: 43840082.2201 - fn0.7: 2251381.9399 - precision0.7: 0.9300 - recall0.7: 0.7245 - tp0.9: 5047421.8314 - fp0.9: 138119.9415 - tn0.9: 44143384.7346 - fn0.9: 3181742.8985 - precision0.9: 0.
  9728 - recall0.9: 0.6102 - accuracy: 0.9528 - auc: 0.9660 - val_loss: 0.8271 - val_tp0.1: 3497398.0000 - val_fp0.1: 2613010.0000 - val_tn0.1: 19230938.0000 - val_fn0.1: 873051.0000 - val_precision0.1: 0.5724 - val_recall0.1: 0.8002 - v
  al_tp0.3: 2805297.0000 - val_fp0.3: 1928525.0000 - val_tn0.3: 19915426.0000 - val_fn0.3: 1565152.0000 - val_precision0.3: 0.5926 - val_recall0.3: 0.6419 - val_tp0.5: 2250608.0000 - val_fp0.5: 1602046.0000 - val_tn0.5: 20241908.0000 - v
  al_fn0.5: 2119841.0000 - val_precision0.5: 0.5842 - val_recall0.5: 0.5150 - val_tp0.7: 1873937.0000 - val_fp0.7: 1289178.0000 - val_tn0.7: 20554764.0000 - val_fn0.7: 2496512.0000 - val_precision0.7: 0.5924 - val_recall0.7: 0.4288 - val
  _tp0.9: 1282739.0000 - val_fp0.9: 663370.0000 - val_tn0.9: 21180588.0000 - val_fn0.9: 3087710.0000 - val_precision0.9: 0.6591 - val_recall0.9: 0.2935 - val_accuracy: 0.8580 - val_auc: 0.8595
  Epoch 11/50
  1280/1280 [==============================] - 3284s 3s/step - loss: 0.3131 - tp0.1: 7805750.0211 - fp0.1: 4180231.1202 - tn0.1: 40107131.8431 - fn0.1: 417539.0390 - precision0.1: 0.6527 - recall0.1: 0.9499 - tp0.3: 7439055.2615 - fp0.3:
   2100484.2233 - tn0.3: 42186897.5113 - fn0.3: 784233.7986 - precision0.3: 0.7813 - recall0.3: 0.9065 - tp0.5: 7021029.5238 - fp0.5: 1158430.3731 - tn0.5: 43128929.4192 - fn0.5: 1202259.5363 - precision0.5: 0.8590 - recall0.5: 0.8568 -
  tp0.7: 6113168.1468 - fp0.7: 426487.4707 - tn0.7: 43860880.4848 - fn0.7: 2110120.9133 - precision0.7: 0.9352 - recall0.7: 0.7472 - tp0.9: 5208463.6464 - fp0.9: 134634.9001 - tn0.9: 44152728.1069 - fn0.9: 3014825.4137 - precision0.9: 0.
  9752 - recall0.9: 0.6369 - accuracy: 0.9554 - auc: 0.9685 - val_loss: 0.8179 - val_tp0.1: 3171244.0000 - val_fp0.1: 2428671.0000 - val_tn0.1: 19710012.0000 - val_fn0.1: 904466.0000 - val_precision0.1: 0.5663 - val_recall0.1: 0.7781 - v
  al_tp0.3: 2577226.0000 - val_fp0.3: 1780010.0000 - val_tn0.3: 20358686.0000 - val_fn0.3: 1498484.0000 - val_precision0.3: 0.5915 - val_recall0.3: 0.6323 - val_tp0.5: 2103216.0000 - val_fp0.5: 1506964.0000 - val_tn0.5: 20631718.0000 - v
  al_fn0.5: 1972494.0000 - val_precision0.5: 0.5826 - val_recall0.5: 0.5160 - val_tp0.7: 1740648.0000 - val_fp0.7: 1184962.0000 - val_tn0.7: 20953724.0000 - val_fn0.7: 2335062.0000 - val_precision0.7: 0.5950 - val_recall0.7: 0.4271 - val
  _tp0.9: 1152892.0000 - val_fp0.9: 479364.0000 - val_tn0.9: 21659326.0000 - val_fn0.9: 2922818.0000 - val_precision0.9: 0.7063 - val_recall0.9: 0.2829 - val_accuracy: 0.8673 - val_auc: 0.8530
  Epoch 12/50
  1280/1280 [==============================] - 3297s 3s/step - loss: 0.3231 - tp0.1: 7673586.0687 - fp0.1: 4254414.5621 - tn0.1: 40148103.3286 - fn0.1: 434544.6292 - precision0.1: 0.6426 - recall0.1: 0.9456 - tp0.3: 7286525.1889 - fp0.3:
   2133646.5324 - tn0.3: 42268881.5730 - fn0.3: 821605.5090 - precision0.3: 0.7727 - recall0.3: 0.8978 - tp0.5: 6849781.8283 - fp0.5: 1164265.2397 - tn0.5: 43238255.9961 - fn0.5: 1258348.8696 - precision0.5: 0.8542 - recall0.5: 0.8438 -
  tp0.7: 5967163.4153 - fp0.7: 423410.0211 - tn0.7: 43979128.0921 - fn0.7: 2140967.2826 - precision0.7: 0.9348 - recall0.7: 0.7339 - tp0.9: 5093586.0039 - fp0.9: 136848.5098 - tn0.9: 44265710.3138 - fn0.9: 3014544.6940 - precision0.9: 0.
  9745 - recall0.9: 0.6262 - accuracy: 0.9537 - auc: 0.9657 - val_loss: 0.7042 - val_tp0.1: 3495859.0000 - val_fp0.1: 2855125.0000 - val_tn0.1: 19280112.0000 - val_fn0.1: 583301.0000 - val_precision0.1: 0.5504 - val_recall0.1: 0.8570 - v
  al_tp0.3: 2938637.0000 - val_fp0.3: 2173835.0000 - val_tn0.3: 19961412.0000 - val_fn0.3: 1140523.0000 - val_precision0.3: 0.5748 - val_recall0.3: 0.7204 - val_tp0.5: 2404430.0000 - val_fp0.5: 1820758.0000 - val_tn0.5: 20314484.0000 - v
  al_fn0.5: 1674730.0000 - val_precision0.5: 0.5691 - val_recall0.5: 0.5894 - val_tp0.7: 2024378.0000 - val_fp0.7: 1527093.0000 - val_tn0.7: 20608140.0000 - val_fn0.7: 2054782.0000 - val_precision0.7: 0.5700 - val_recall0.7: 0.4963 - val
  _tp0.9: 1381535.0000 - val_fp0.9: 912692.0000 - val_tn0.9: 21222556.0000 - val_fn0.9: 2697625.0000 - val_precision0.9: 0.6022 - val_recall0.9: 0.3387 - val_accuracy: 0.8667 - val_auc: 0.8805
  Epoch 13/50
  1280/1280 [==============================] - 3321s 3s/step - loss: 0.3169 - tp0.1: 7909004.5371 - fp0.1: 4146765.2818 - tn0.1: 40030494.6401 - fn0.1: 424405.8938 - precision0.1: 0.6543 - recall0.1: 0.9500 - tp0.3: 7536643.9977 - fp0.3:
   2085465.0749 - tn0.3: 42091771.2584 - fn0.3: 796766.4333 - precision0.3: 0.7808 - recall0.3: 0.9063 - tp0.5: 7106355.0859 - fp0.5: 1146387.5933 - tn0.5: 43030851.5699 - fn0.5: 1227055.3450 - precision0.5: 0.8590 - recall0.5: 0.8551 -
  tp0.7: 6229305.1702 - fp0.7: 429169.7603 - tn0.7: 43748120.6526 - fn0.7: 2104105.2607 - precision0.7: 0.9350 - recall0.7: 0.7476 - tp0.9: 5365662.5207 - fp0.9: 142482.5558 - tn0.9: 44034765.0882 - fn0.9: 2967747.9102 - precision0.9: 0.
  9736 - recall0.9: 0.6436 - accuracy: 0.9544 - auc: 0.9678 - val_loss: 0.8398 - val_tp0.1: 3245885.0000 - val_fp0.1: 2536432.0000 - val_tn0.1: 19519276.0000 - val_fn0.1: 912810.0000 - val_precision0.1: 0.5613 - val_recall0.1: 0.7805 - v
  al_tp0.3: 2564525.0000 - val_fp0.3: 1814112.0000 - val_tn0.3: 20241596.0000 - val_fn0.3: 1594170.0000 - val_precision0.3: 0.5857 - val_recall0.3: 0.6167 - val_tp0.5: 2015465.0000 - val_fp0.5: 1451082.0000 - val_tn0.5: 20604624.0000 - v
  al_fn0.5: 2143230.0000 - val_precision0.5: 0.5814 - val_recall0.5: 0.4846 - val_tp0.7: 1621561.0000 - val_fp0.7: 1053037.0000 - val_tn0.7: 21002666.0000 - val_fn0.7: 2537134.0000 - val_precision0.7: 0.6063 - val_recall0.7: 0.3899 - val
  _tp0.9: 1065568.0000 - val_fp0.9: 460390.0000 - val_tn0.9: 21595312.0000 - val_fn0.9: 3093127.0000 - val_precision0.9: 0.6983 - val_recall0.9: 0.2562 - val_accuracy: 0.8629 - val_auc: 0.8493
  Epoch 14/50
  1280/1280 [==============================] - 3314s 3s/step - loss: 0.3076 - tp0.1: 7765395.1202 - fp0.1: 4130752.8431 - tn0.1: 40213352.2810 - fn0.1: 401143.6737 - precision0.1: 0.6501 - recall0.1: 0.9503 - tp0.3: 7389168.6144 - fp0.3:
   2041484.8189 - tn0.3: 42302624.6870 - fn0.3: 777370.1795 - precision0.3: 0.7825 - recall0.3: 0.9030 - tp0.5: 6961617.5222 - fp0.5: 1091504.5597 - tn0.5: 43252606.1101 - fn0.5: 1204921.2717 - precision0.5: 0.8642 - recall0.5: 0.8492 -
  tp0.7: 6118152.3286 - fp0.7: 416044.1280 - tn0.7: 43928097.5558 - fn0.7: 2048386.4653 - precision0.7: 0.9364 - recall0.7: 0.7464 - tp0.9: 5275694.6081 - fp0.9: 131773.3169 - tn0.9: 44212345.9774 - fn0.9: 2890844.1858 - precision0.9: 0.
  9759 - recall0.9: 0.6429 - accuracy: 0.9559 - auc: 0.9681 - val_loss: 0.7646 - val_tp0.1: 3394116.0000 - val_fp0.1: 2434359.0000 - val_tn0.1: 19620386.0000 - val_fn0.1: 765541.0000 - val_precision0.1: 0.5823 - val_recall0.1: 0.8160 - v
  al_tp0.3: 2854698.0000 - val_fp0.3: 1867547.0000 - val_tn0.3: 20187194.0000 - val_fn0.3: 1304959.0000 - val_precision0.3: 0.6045 - val_recall0.3: 0.6863 - val_tp0.5: 2392164.0000 - val_fp0.5: 1600876.0000 - val_tn0.5: 20453860.0000 - v
  al_fn0.5: 1767493.0000 - val_precision0.5: 0.5991 - val_recall0.5: 0.5751 - val_tp0.7: 2014352.0000 - val_fp0.7: 1325938.0000 - val_tn0.7: 20728804.0000 - val_fn0.7: 2145305.0000 - val_precision0.7: 0.6030 - val_recall0.7: 0.4843 - val
  _tp0.9: 1382581.0000 - val_fp0.9: 710524.0000 - val_tn0.9: 21344216.0000 - val_fn0.9: 2777076.0000 - val_precision0.9: 0.6605 - val_recall0.9: 0.3324 - val_accuracy: 0.8715 - val_auc: 0.8680
  Epoch 15/50
  1280/1280 [==============================] - 3321s 3s/step - loss: 0.3012 - tp0.1: 7918202.5168 - fp0.1: 3944007.1944 - tn0.1: 40234677.5113 - fn0.1: 413802.1538 - precision0.1: 0.6678 - recall0.1: 0.9509 - tp0.3: 7545589.1913 - fp0.3:
   1942385.4692 - tn0.3: 42236259.0687 - fn0.3: 786415.4793 - precision0.3: 0.7966 - recall0.3: 0.9059 - tp0.5: 7139685.9657 - fp0.5: 1075216.9391 - tn0.5: 43103450.5808 - fn0.5: 1192318.7049 - precision0.5: 0.8712 - recall0.5: 0.8570 -
  tp0.7: 6377161.2748 - fp0.7: 450450.9469 - tn0.7: 43728240.0265 - fn0.7: 1954843.3958 - precision0.7: 0.9369 - recall0.7: 0.7641 - tp0.9: 5506362.6120 - fp0.9: 143446.8002 - tn0.9: 44035203.3466 - fn0.9: 2825642.0585 - precision0.9: 0.
  9752 - recall0.9: 0.6625 - accuracy: 0.9571 - auc: 0.9692 - val_loss: 0.7740 - val_tp0.1: 3355047.0000 - val_fp0.1: 2511597.0000 - val_tn0.1: 19571600.0000 - val_fn0.1: 776157.0000 - val_precision0.1: 0.5719 - val_recall0.1: 0.8121 - v
  al_tp0.3: 2793785.0000 - val_fp0.3: 1928920.0000 - val_tn0.3: 20154280.0000 - val_fn0.3: 1337419.0000 - val_precision0.3: 0.5916 - val_recall0.3: 0.6763 - val_tp0.5: 2272116.0000 - val_fp0.5: 1578770.0000 - val_tn0.5: 20504436.0000 - v
  al_fn0.5: 1859088.0000 - val_precision0.5: 0.5900 - val_recall0.5: 0.5500 - val_tp0.7: 1883335.0000 - val_fp0.7: 1254872.0000 - val_tn0.7: 20828326.0000 - val_fn0.7: 2247869.0000 - val_precision0.7: 0.6001 - val_recall0.7: 0.4559 - val
  _tp0.9: 1294491.0000 - val_fp0.9: 656247.0000 - val_tn0.9: 21426948.0000 - val_fn0.9: 2836713.0000 - val_precision0.9: 0.6636 - val_recall0.9: 0.3133 - val_accuracy: 0.8689 - val_auc: 0.8643
  Epoch 16/50
  1280/1280 [==============================] - 3322s 3s/step - loss: 0.2996 - tp0.1: 7939368.5972 - fp0.1: 3969169.3778 - tn0.1: 40211699.6347 - fn0.1: 390406.3911 - precision0.1: 0.6606 - recall0.1: 0.9528 - tp0.3: 7573992.7580 - fp0.3:
   1993073.9243 - tn0.3: 42187811.5020 - fn0.3: 755782.2303 - precision0.3: 0.7866 - recall0.3: 0.9074 - tp0.5: 7177194.8970 - fp0.5: 1109584.0390 - tn0.5: 43071285.9805 - fn0.5: 1152580.0913 - precision0.5: 0.8632 - recall0.5: 0.8586 -
  tp0.7: 6618657.6151 - fp0.7: 578836.1054 - tn0.7: 43602043.0445 - fn0.7: 1711117.3731 - precision0.7: 0.9188 - recall0.7: 0.7896 - tp0.9: 5482513.6440 - fp0.9: 141140.2100 - tn0.9: 44039753.1483 - fn0.9: 2847261.3443 - precision0.9: 0.
  9744 - recall0.9: 0.6538 - accuracy: 0.9565 - auc: 0.9693 - val_loss: 0.8054 - val_tp0.1: 3345481.0000 - val_fp0.1: 2254991.0000 - val_tn0.1: 19696408.0000 - val_fn0.1: 917515.0000 - val_precision0.1: 0.5974 - val_recall0.1: 0.7848 - v
  al_tp0.3: 2726884.0000 - val_fp0.3: 1640058.0000 - val_tn0.3: 20311344.0000 - val_fn0.3: 1536112.0000 - val_precision0.3: 0.6244 - val_recall0.3: 0.6397 - val_tp0.5: 2263115.0000 - val_fp0.5: 1402171.0000 - val_tn0.5: 20549236.0000 - v
  al_fn0.5: 1999881.0000 - val_precision0.5: 0.6174 - val_recall0.5: 0.5309 - val_tp0.7: 1906416.0000 - val_fp0.7: 1106825.0000 - val_tn0.7: 20844576.0000 - val_fn0.7: 2356580.0000 - val_precision0.7: 0.6327 - val_recall0.7: 0.4472 - val
  _tp0.9: 1297258.0000 - val_fp0.9: 495212.0000 - val_tn0.9: 21456192.0000 - val_fn0.9: 2965738.0000 - val_precision0.9: 0.7237 - val_recall0.9: 0.3043 - val_accuracy: 0.8702 - val_auc: 0.8614
  Epoch 17/50
  1280/1280 [==============================] - 3281s 3s/step - loss: 0.2900 - tp0.1: 7724454.9820 - fp0.1: 3793475.2631 - tn0.1: 40600766.8860 - fn0.1: 391947.2693 - precision0.1: 0.6695 - recall0.1: 0.9516 - tp0.3: 7368504.9789 - fp0.3:
   1879500.1913 - tn0.3: 42514764.5855 - fn0.3: 747897.2724 - precision0.3: 0.7960 - recall0.3: 0.9078 - tp0.5: 6984976.5464 - fp0.5: 1046817.2740 - tn0.5: 43347456.7892 - fn0.5: 1131425.7049 - precision0.5: 0.8690 - recall0.5: 0.8606 -
  tp0.7: 6474243.4176 - fp0.7: 552184.5105 - tn0.7: 43842053.0726 - fn0.7: 1642158.8337 - precision0.7: 0.9206 - recall0.7: 0.7983 - tp0.9: 5438164.0788 - fp0.9: 137406.8056 - tn0.9: 44256829.4044 - fn0.9: 2678238.1725 - precision0.9: 0.
  9752 - recall0.9: 0.6718 - accuracy: 0.9584 - auc: 0.9698 - val_loss: 0.8475 - val_tp0.1: 3212091.0000 - val_fp0.1: 2556630.0000 - val_tn0.1: 19565418.0000 - val_fn0.1: 880263.0000 - val_precision0.1: 0.5568 - val_recall0.1: 0.7849 - v
  al_tp0.3: 2676698.0000 - val_fp0.3: 1975975.0000 - val_tn0.3: 20146070.0000 - val_fn0.3: 1415656.0000 - val_precision0.3: 0.5753 - val_recall0.3: 0.6541 - val_tp0.5: 2228667.0000 - val_fp0.5: 1724443.0000 - val_tn0.5: 20397608.0000 - v
  al_fn0.5: 1863687.0000 - val_precision0.5: 0.5638 - val_recall0.5: 0.5446 - val_tp0.7: 1876988.0000 - val_fp0.7: 1421139.0000 - val_tn0.7: 20700906.0000 - val_fn0.7: 2215366.0000 - val_precision0.7: 0.5691 - val_recall0.7: 0.4587 - val
  _tp0.9: 1318523.0000 - val_fp0.9: 744640.0000 - val_tn0.9: 21377414.0000 - val_fn0.9: 2773831.0000 - val_precision0.9: 0.6391 - val_recall0.9: 0.3222 - val_accuracy: 0.8631 - val_auc: 0.8479
  Epoch 18/50
  1280/1280 [==============================] - 3291s 3s/step - loss: 0.2960 - tp0.1: 7878258.3583 - fp0.1: 3966414.4723 - tn0.1: 40266792.8720 - fn0.1: 399191.5441 - precision0.1: 0.6660 - recall0.1: 0.9518 - tp0.3: 7519728.0133 - fp0.3:
   1997387.8189 - tn0.3: 42235815.4325 - fn0.3: 757721.8891 - precision0.3: 0.7910 - recall0.3: 0.9083 - tp0.5: 7117268.5543 - fp0.5: 1103594.4309 - tn0.5: 43129603.3357 - fn0.5: 1160181.3482 - precision0.5: 0.8660 - recall0.5: 0.8598 -
  tp0.7: 6559720.6885 - fp0.7: 569817.0601 - tn0.7: 43663373.8767 - fn0.7: 1717729.2139 - precision0.7: 0.9203 - recall0.7: 0.7929 - tp0.9: 5460392.9578 - fp0.9: 133811.0039 - tn0.9: 44099404.8353 - fn0.9: 2817056.9446 - precision0.9: 0.
  9765 - recall0.9: 0.6620 - accuracy: 0.9569 - auc: 0.9692 - val_loss: 2.4700 - val_tp0.1: 3478680.0000 - val_fp0.1: 2735141.0000 - val_tn0.1: 19398832.0000 - val_fn0.1: 601742.0000 - val_precision0.1: 0.5598 - val_recall0.1: 0.8525 - v
  al_tp0.3: 2990828.0000 - val_fp0.3: 2147854.0000 - val_tn0.3: 19986120.0000 - val_fn0.3: 1089594.0000 - val_precision0.3: 0.5820 - val_recall0.3: 0.7330 - val_tp0.5: 2523597.0000 - val_fp0.5: 1960329.0000 - val_tn0.5: 20173640.0000 - v
  al_fn0.5: 1556825.0000 - val_precision0.5: 0.5628 - val_recall0.5: 0.6185 - val_tp0.7: 2211753.0000 - val_fp0.7: 1837061.0000 - val_tn0.7: 20296910.0000 - val_fn0.7: 1868669.0000 - val_precision0.7: 0.5463 - val_recall0.7: 0.5420 - val
  _tp0.9: 1781271.0000 - val_fp0.9: 1591848.0000 - val_tn0.9: 20542120.0000 - val_fn0.9: 2299151.0000 - val_precision0.9: 0.5281 - val_recall0.9: 0.4365 - val_accuracy: 0.8658 - val_auc: 0.8707
  Epoch 19/50
  1280/1280 [==============================] - 3287s 3s/step - loss: 0.2909 - tp0.1: 7989303.7198 - fp0.1: 3862557.4653 - tn0.1: 40273532.0609 - fn0.1: 385255.3396 - precision0.1: 0.6759 - recall0.1: 0.9542 - tp0.3: 7630793.7799 - fp0.3:
   1926931.7330 - tn0.3: 42209171.3419 - fn0.3: 743765.2795 - precision0.3: 0.7994 - recall0.3: 0.9113 - tp0.5: 7240582.7869 - fp0.5: 1067331.3224 - tn0.5: 43068761.2030 - fn0.5: 1133976.2724 - precision0.5: 0.8729 - recall0.5: 0.8645 -
  tp0.7: 6700153.0500 - fp0.7: 553561.1038 - tn0.7: 43582525.7065 - fn0.7: 1674406.0094 - precision0.7: 0.9247 - recall0.7: 0.7993 - tp0.9: 5619106.6877 - fp0.9: 138913.2576 - tn0.9: 43997191.2326 - fn0.9: 2755452.3716 - precision0.9: 0.
  9761 - recall0.9: 0.6706 - accuracy: 0.9580 - auc: 0.9708 - val_loss: 0.8749 - val_tp0.1: 3092711.0000 - val_fp0.1: 2067604.0000 - val_tn0.1: 20017088.0000 - val_fn0.1: 1036989.0000 - val_precision0.1: 0.5993 - val_recall0.1: 0.7489 -
  val_tp0.3: 2486761.0000 - val_fp0.3: 1549670.0000 - val_tn0.3: 20535032.0000 - val_fn0.3: 1642939.0000 - val_precision0.3: 0.6161 - val_recall0.3: 0.6022 - val_tp0.5: 2068354.0000 - val_fp0.5: 1334103.0000 - val_tn0.5: 20750600.0000 -
  val_fn0.5: 2061346.0000 - val_precision0.5: 0.6079 - val_recall0.5: 0.5008 - val_tp0.7: 1728814.0000 - val_fp0.7: 1041223.0000 - val_tn0.7: 21043472.0000 - val_fn0.7: 2400886.0000 - val_precision0.7: 0.6241 - val_recall0.7: 0.4186 - va
  l_tp0.9: 1199934.0000 - val_fp0.9: 520759.0000 - val_tn0.9: 21563940.0000 - val_fn0.9: 2929766.0000 - val_precision0.9: 0.6974 - val_recall0.9: 0.2906 - val_accuracy: 0.8705 - val_auc: 0.8436
  Epoch 20/50
  1280/1280 [==============================] - 3278s 3s/step - loss: 0.2874 - tp0.1: 7845801.3482 - fp0.1: 3747571.3029 - tn0.1: 40520331.9508 - fn0.1: 396960.4161 - precision0.1: 0.6782 - recall0.1: 0.9514 - tp0.3: 7499855.0742 - fp0.3:
   1875299.3388 - tn0.3: 42392596.7424 - fn0.3: 742906.6901 - precision0.3: 0.8017 - recall0.3: 0.9098 - tp0.5: 7121660.4918 - fp0.5: 1042386.0788 - tn0.5: 43225501.8384 - fn0.5: 1121101.2724 - precision0.5: 0.8739 - recall0.5: 0.8640 -
  tp0.7: 6592531.3404 - fp0.7: 538088.3911 - tn0.7: 43729809.0500 - fn0.7: 1650230.4239 - precision0.7: 0.9254 - recall0.7: 0.8003 - tp0.9: 5543415.5105 - fp0.9: 131938.6378 - tn0.9: 44135942.6190 - fn0.9: 2699346.2537 - precision0.9: 0.
  9770 - recall0.9: 0.6728 - accuracy: 0.9592 - auc: 0.9698 - val_loss: 0.9565 - val_tp0.1: 2943258.0000 - val_fp0.1: 2086336.0000 - val_tn0.1: 20045976.0000 - val_fn0.1: 1138833.0000 - val_precision0.1: 0.5852 - val_recall0.1: 0.7210 -
  val_tp0.3: 2419480.0000 - val_fp0.3: 1662989.0000 - val_tn0.3: 20469314.0000 - val_fn0.3: 1662611.0000 - val_precision0.3: 0.5927 - val_recall0.3: 0.5927 - val_tp0.5: 2018052.0000 - val_fp0.5: 1478186.0000 - val_tn0.5: 20654124.0000 -
  val_fn0.5: 2064039.0000 - val_precision0.5: 0.5772 - val_recall0.5: 0.4944 - val_tp0.7: 1715248.0000 - val_fp0.7: 1231354.0000 - val_tn0.7: 20900956.0000 - val_fn0.7: 2366843.0000 - val_precision0.7: 0.5821 - val_recall0.7: 0.4202 - va
  l_tp0.9: 1245570.0000 - val_fp0.9: 677868.0000 - val_tn0.9: 21454434.0000 - val_fn0.9: 2836521.0000 - val_precision0.9: 0.6476 - val_recall0.9: 0.3051 - val_accuracy: 0.8649 - val_auc: 0.8275
  Epoch 21/50
  1280/1280 [==============================] - 3290s 3s/step - loss: 0.2804 - tp0.1: 7821343.4738 - fp0.1: 3664726.1171 - tn0.1: 40657232.2397 - fn0.1: 367350.5347 - precision0.1: 0.6815 - recall0.1: 0.9542 - tp0.3: 7489144.7580 - fp0.3:
   1868735.6292 - tn0.3: 42453227.8447 - fn0.3: 699549.2506 - precision0.3: 0.8010 - recall0.3: 0.9132 - tp0.5: 7115665.6729 - fp0.5: 1043113.8876 - tn0.5: 43278844.9688 - fn0.5: 1073028.3357 - precision0.5: 0.8724 - recall0.5: 0.8679 -
  tp0.7: 6602597.8665 - fp0.7: 551623.0804 - tn0.7: 43770320.6628 - fn0.7: 1586096.1421 - precision0.7: 0.9226 - recall0.7: 0.8053 - tp0.9: 5554093.7018 - fp0.9: 134091.5519 - tn0.9: 44187854.0671 - fn0.9: 2634600.3068 - precision0.9: 0.
  9763 - recall0.9: 0.6772 - accuracy: 0.9595 - auc: 0.9714 - val_loss: 0.8160 - val_tp0.1: 3192064.0000 - val_fp0.1: 2182394.0000 - val_tn0.1: 19900398.0000 - val_fn0.1: 939537.0000 - val_precision0.1: 0.5939 - val_recall0.1: 0.7726 - v
  al_tp0.3: 2628251.0000 - val_fp0.3: 1676252.0000 - val_tn0.3: 20406544.0000 - val_fn0.3: 1503350.0000 - val_precision0.3: 0.6106 - val_recall0.3: 0.6361 - val_tp0.5: 2262964.0000 - val_fp0.5: 1459711.0000 - val_tn0.5: 20623092.0000 - v
  al_fn0.5: 1868637.0000 - val_precision0.5: 0.6079 - val_recall0.5: 0.5477 - val_tp0.7: 1912004.0000 - val_fp0.7: 1114695.0000 - val_tn0.7: 20968108.0000 - val_fn0.7: 2219597.0000 - val_precision0.7: 0.6317 - val_recall0.7: 0.4628 - val
  _tp0.9: 1266412.0000 - val_fp0.9: 537157.0000 - val_tn0.9: 21545640.0000 - val_fn0.9: 2865189.0000 - val_precision0.9: 0.7022 - val_recall0.9: 0.3065 - val_accuracy: 0.8730 - val_auc: 0.8570
  Epoch 22/50
  1280/1280 [==============================] - 3337s 3s/step - loss: 0.2695 - tp0.1: 7964599.1304 - fp0.1: 3487980.5644 - tn0.1: 40692123.6760 - fn0.1: 365971.2584 - precision0.1: 0.6952 - recall0.1: 0.9559 - tp0.3: 7639300.1928 - fp0.3:
   1768977.4333 - tn0.3: 42411101.8767 - fn0.3: 691270.1959 - precision0.3: 0.8119 - recall0.3: 0.9169 - tp0.5: 7283022.7681 - fp0.5: 1008769.9313 - tn0.5: 43171283.5621 - fn0.5: 1047547.6206 - precision0.5: 0.8781 - recall0.5: 0.8740 -
  tp0.7: 6797706.8946 - fp0.7: 543031.7963 - tn0.7: 43637044.5636 - fn0.7: 1532863.4941 - precision0.7: 0.9257 - recall0.7: 0.8158 - tp0.9: 5748924.9516 - fp0.9: 136050.2350 - tn0.9: 44044056.0921 - fn0.9: 2581645.4372 - precision0.9: 0.
  9771 - recall0.9: 0.6893 - accuracy: 0.9609 - auc: 0.9731 - val_loss: 0.8131 - val_tp0.1: 3187368.0000 - val_fp0.1: 2205558.0000 - val_tn0.1: 19939310.0000 - val_fn0.1: 882169.0000 - val_precision0.1: 0.5910 - val_recall0.1: 0.7832 - v
  al_tp0.3: 2568349.0000 - val_fp0.3: 1630822.0000 - val_tn0.3: 20514044.0000 - val_fn0.3: 1501188.0000 - val_precision0.3: 0.6116 - val_recall0.3: 0.6311 - val_tp0.5: 2132479.0000 - val_fp0.5: 1391371.0000 - val_tn0.5: 20753492.0000 - v
  al_fn0.5: 1937058.0000 - val_precision0.5: 0.6052 - val_recall0.5: 0.5240 - val_tp0.7: 1724396.0000 - val_fp0.7: 1052743.0000 - val_tn0.7: 21092120.0000 - val_fn0.7: 2345141.0000 - val_precision0.7: 0.6209 - val_recall0.7: 0.4237 - val
  _tp0.9: 1088044.0000 - val_fp0.9: 509365.0000 - val_tn0.9: 21635508.0000 - val_fn0.9: 2981493.0000 - val_precision0.9: 0.6811 - val_recall0.9: 0.2674 - val_accuracy: 0.8730 - val_auc: 0.8564
  Epoch 23/50
  1280/1280 [==============================] - 3288s 3s/step - loss: 0.2689 - tp0.1: 7750995.1733 - fp0.1: 3597605.4887 - tn0.1: 40799177.1624 - fn0.1: 362898.6737 - precision0.1: 0.6830 - recall0.1: 0.9562 - tp0.3: 7415912.0773 - fp0.3:
   1777617.9633 - tn0.3: 42619150.5051 - fn0.3: 697981.7697 - precision0.3: 0.8063 - recall0.3: 0.9146 - tp0.5: 7059667.4192 - fp0.5: 992741.5051 - tn0.5: 43404008.9727 - fn0.5: 1054226.4278 - precision0.5: 0.8769 - recall0.5: 0.8705 - t
  p0.7: 6579822.4957 - fp0.7: 525915.9500 - tn0.7: 43870813.2443 - fn0.7: 1534071.3513 - precision0.7: 0.9263 - recall0.7: 0.8112 - tp0.9: 5573469.2935 - fp0.9: 125489.0593 - tn0.9: 44271266.5379 - fn0.9: 2540424.5535 - precision0.9: 0.9
  785 - recall0.9: 0.6878 - accuracy: 0.9612 - auc: 0.9727 - val_loss: 0.8991 - val_tp0.1: 3067116.0000 - val_fp0.1: 2378685.0000 - val_tn0.1: 19772492.0000 - val_fn0.1: 996109.0000 - val_precision0.1: 0.5632 - val_recall0.1: 0.7548 - va
  l_tp0.3: 2525982.0000 - val_fp0.3: 1869663.0000 - val_tn0.3: 20281514.0000 - val_fn0.3: 1537243.0000 - val_precision0.3: 0.5747 - val_recall0.3: 0.6217 - val_tp0.5: 2090818.0000 - val_fp0.5: 1570273.0000 - val_tn0.5: 20580908.0000 - va
  l_fn0.5: 1972407.0000 - val_precision0.5: 0.5711 - val_recall0.5: 0.5146 - val_tp0.7: 1691191.0000 - val_fp0.7: 1177904.0000 - val_tn0.7: 20973274.0000 - val_fn0.7: 2372034.0000 - val_precision0.7: 0.5895 - val_recall0.7: 0.4162 - val_
  tp0.9: 1101527.0000 - val_fp0.9: 574543.0000 - val_tn0.9: 21576624.0000 - val_fn0.9: 2961698.0000 - val_precision0.9: 0.6572 - val_recall0.9: 0.2711 - val_accuracy: 0.8649 - val_auc: 0.8377
  Epoch 24/50
  1280/1280 [==============================] - 3284s 3s/step - loss: 0.2805 - tp0.1: 7918656.6589 - fp0.1: 3592495.9235 - tn0.1: 40635399.4606 - fn0.1: 364123.3185 - precision0.1: 0.6839 - recall0.1: 0.9541 - tp0.3: 7581799.2061 - fp0.3:
   1810020.2311 - tn0.3: 42417852.9594 - fn0.3: 700980.7713 - precision0.3: 0.8032 - recall0.3: 0.9122 - tp0.5: 7211523.7502 - fp0.5: 1027547.9930 - tn0.5: 43200324.1296 - fn0.5: 1071256.2272 - precision0.5: 0.8718 - recall0.5: 0.8665 -
  tp0.7: 6727091.3692 - fp0.7: 558955.8712 - tn0.7: 43668944.4075 - fn0.7: 1555688.6081 - precision0.7: 0.9209 - recall0.7: 0.8072 - tp0.9: 5696121.9555 - fp0.9: 142622.7455 - tn0.9: 44085280.0343 - fn0.9: 2586658.0219 - precision0.9: 0.
  9746 - recall0.9: 0.6826 - accuracy: 0.9591 - auc: 0.9708 - val_loss: 0.9386 - val_tp0.1: 2779932.0000 - val_fp0.1: 1939286.0000 - val_tn0.1: 20344936.0000 - val_fn0.1: 1150243.0000 - val_precision0.1: 0.5891 - val_recall0.1: 0.7073 -
  val_tp0.3: 2181594.0000 - val_fp0.3: 1457307.0000 - val_tn0.3: 20826908.0000 - val_fn0.3: 1748581.0000 - val_precision0.3: 0.5995 - val_recall0.3: 0.5551 - val_tp0.5: 1772386.0000 - val_fp0.5: 1197130.0000 - val_tn0.5: 21087092.0000 -
  val_fn0.5: 2157789.0000 - val_precision0.5: 0.5969 - val_recall0.5: 0.4510 - val_tp0.7: 1397374.0000 - val_fp0.7: 840833.0000 - val_tn0.7: 21443384.0000 - val_fn0.7: 2532801.0000 - val_precision0.7: 0.6243 - val_recall0.7: 0.3556 - val
  _tp0.9: 891892.0000 - val_fp0.9: 338054.0000 - val_tn0.9: 21946184.0000 - val_fn0.9: 3038283.0000 - val_precision0.9: 0.7251 - val_recall0.9: 0.2269 - val_accuracy: 0.8720 - val_auc: 0.8227
  Epoch 25/50
  1280/1280 [==============================] - 3312s 3s/step - loss: 0.2614 - tp0.1: 7920216.7338 - fp0.1: 3526618.6542 - tn0.1: 40723751.9001 - fn0.1: 340058.4512 - precision0.1: 0.6948 - recall0.1: 0.9594 - tp0.3: 7591746.0851 - fp0.3:
   1770255.0687 - tn0.3: 42480125.7884 - fn0.3: 668529.0999 - precision0.3: 0.8133 - recall0.3: 0.9199 - tp0.5: 7232148.8064 - fp0.5: 1004410.7486 - tn0.5: 43245980.8556 - fn0.5: 1028126.3786 - precision0.5: 0.8800 - recall0.5: 0.8766 -
  tp0.7: 6751684.1514 - fp0.7: 540161.4325 - tn0.7: 43710214.4629 - fn0.7: 1508591.0336 - precision0.7: 0.9275 - recall0.7: 0.8179 - tp0.9: 5682987.5933 - fp0.9: 133426.7330 - tn0.9: 44116927.1835 - fn0.9: 2577287.5917 - precision0.9: 0.
  9781 - recall0.9: 0.6873 - accuracy: 0.9617 - auc: 0.9742 - val_loss: 0.7990 - val_tp0.1: 3208335.0000 - val_fp0.1: 2291707.0000 - val_tn0.1: 19888704.0000 - val_fn0.1: 825651.0000 - val_precision0.1: 0.5833 - val_recall0.1: 0.7953 - v
  al_tp0.3: 2646324.0000 - val_fp0.3: 1812815.0000 - val_tn0.3: 20367606.0000 - val_fn0.3: 1387662.0000 - val_precision0.3: 0.5935 - val_recall0.3: 0.6560 - val_tp0.5: 2184535.0000 - val_fp0.5: 1449172.0000 - val_tn0.5: 20731252.0000 - v
  al_fn0.5: 1849451.0000 - val_precision0.5: 0.6012 - val_recall0.5: 0.5415 - val_tp0.7: 1700807.0000 - val_fp0.7: 1059055.0000 - val_tn0.7: 21121356.0000 - val_fn0.7: 2333179.0000 - val_precision0.7: 0.6163 - val_recall0.7: 0.4216 - val
  _tp0.9: 1000799.0000 - val_fp0.9: 603281.0000 - val_tn0.9: 21577132.0000 - val_fn0.9: 3033187.0000 - val_precision0.9: 0.6239 - val_recall0.9: 0.2481 - val_accuracy: 0.8742 - val_auc: 0.8559
  Epoch 26/50
  1280/1280 [==============================] - 3292s 3s/step - loss: 0.2564 - tp0.1: 7951938.6698 - fp0.1: 3507608.0250 - tn0.1: 40722855.8618 - fn0.1: 328256.7658 - precision0.1: 0.6956 - recall0.1: 0.9610 - tp0.3: 7628245.6300 - fp0.3:
   1764444.8579 - tn0.3: 42466010.5285 - fn0.3: 651949.8056 - precision0.3: 0.8130 - recall0.3: 0.9222 - tp0.5: 7277018.4473 - fp0.5: 993563.1429 - tn0.5: 43236890.8002 - fn0.5: 1003176.9883 - precision0.5: 0.8801 - recall0.5: 0.8802 - t
  p0.7: 6809211.9329 - fp0.7: 538938.6565 - tn0.7: 43691507.5847 - fn0.7: 1470983.5027 - precision0.7: 0.9271 - recall0.7: 0.8247 - tp0.9: 5778468.3435 - fp0.9: 130088.7877 - tn0.9: 44100381.7361 - fn0.9: 2501727.0921 - precision0.9: 0.9
  783 - recall0.9: 0.7006 - accuracy: 0.9620 - auc: 0.9752 - val_loss: 0.9080 - val_tp0.1: 2917153.0000 - val_fp0.1: 2217207.0000 - val_tn0.1: 20036056.0000 - val_fn0.1: 1043985.0000 - val_precision0.1: 0.5682 - val_recall0.1: 0.7364 - v
  al_tp0.3: 2324132.0000 - val_fp0.3: 1709552.0000 - val_tn0.3: 20543718.0000 - val_fn0.3: 1637006.0000 - val_precision0.3: 0.5762 - val_recall0.3: 0.5867 - val_tp0.5: 1885949.0000 - val_fp0.5: 1373092.0000 - val_tn0.5: 20880176.0000 - v
  al_fn0.5: 2075189.0000 - val_precision0.5: 0.5787 - val_recall0.5: 0.4761 - val_tp0.7: 1470934.0000 - val_fp0.7: 951612.0000 - val_tn0.7: 21301648.0000 - val_fn0.7: 2490204.0000 - val_precision0.7: 0.6072 - val_recall0.7: 0.3713 - val_
  tp0.9: 925487.0000 - val_fp0.9: 440991.0000 - val_tn0.9: 21812280.0000 - val_fn0.9: 3035651.0000 - val_precision0.9: 0.6773 - val_recall0.9: 0.2336 - val_accuracy: 0.8685 - val_auc: 0.8286
  Epoch 27/50
  1280/1280 [==============================] - 3294s 3s/step - loss: 0.2645 - tp0.1: 8005473.6409 - fp0.1: 3575769.4098 - tn0.1: 40581931.1319 - fn0.1: 347472.6667 - precision0.1: 0.6907 - recall0.1: 0.9587 - tp0.3: 7670517.9243 - fp0.3:
   1798604.7674 - tn0.3: 42359111.5941 - fn0.3: 682428.3833 - precision0.3: 0.8108 - recall0.3: 0.9185 - tp0.5: 7312500.5558 - fp0.5: 1024504.0812 - tn0.5: 43133203.4434 - fn0.5: 1040445.7518 - precision0.5: 0.8784 - recall0.5: 0.8764 -
  tp0.7: 6846122.2201 - fp0.7: 554274.4294 - tn0.7: 43603409.3521 - fn0.7: 1506824.0874 - precision0.7: 0.9264 - recall0.7: 0.8215 - tp0.9: 5804102.2420 - fp0.9: 127947.9672 - tn0.9: 44029764.9969 - fn0.9: 2548844.0656 - precision0.9: 0.
  9789 - recall0.9: 0.6991 - accuracy: 0.9613 - auc: 0.9737 - val_loss: 0.7467 - val_tp0.1: 3265250.0000 - val_fp0.1: 2199966.0000 - val_tn0.1: 19968754.0000 - val_fn0.1: 780424.0000 - val_precision0.1: 0.5975 - val_recall0.1: 0.8071 - v
  al_tp0.3: 2595986.0000 - val_fp0.3: 1569977.0000 - val_tn0.3: 20598752.0000 - val_fn0.3: 1449688.0000 - val_precision0.3: 0.6231 - val_recall0.3: 0.6417 - val_tp0.5: 2090775.0000 - val_fp0.5: 1320764.0000 - val_tn0.5: 20847968.0000 - v
  al_fn0.5: 1954899.0000 - val_precision0.5: 0.6129 - val_recall0.5: 0.5168 - val_tp0.7: 1610859.0000 - val_fp0.7: 895071.0000 - val_tn0.7: 21273662.0000 - val_fn0.7: 2434815.0000 - val_precision0.7: 0.6428 - val_recall0.7: 0.3982 - val_
  tp0.9: 934620.0000 - val_fp0.9: 380349.0000 - val_tn0.9: 21788384.0000 - val_fn0.9: 3111054.0000 - val_precision0.9: 0.7108 - val_recall0.9: 0.2310 - val_accuracy: 0.8750 - val_auc: 0.8659
  Epoch 28/50
  1280/1280 [==============================] - 3286s 3s/step - loss: 0.2563 - tp0.1: 7900461.9563 - fp0.1: 3575374.1405 - tn0.1: 40711544.3599 - fn0.1: 323306.9867 - precision0.1: 0.6896 - recall0.1: 0.9612 - tp0.3: 7567966.7549 - fp0.3:
   1768843.2389 - tn0.3: 42518056.5605 - fn0.3: 655802.1881 - precision0.3: 0.8113 - recall0.3: 0.9206 - tp0.5: 7202561.9477 - fp0.5: 988935.1319 - tn0.5: 43297946.7713 - fn0.5: 1021206.9953 - precision0.5: 0.8795 - recall0.5: 0.8766 - t
  p0.7: 6725537.1077 - fp0.7: 525482.0679 - tn0.7: 43761380.4372 - fn0.7: 1498231.8353 - precision0.7: 0.9278 - recall0.7: 0.8191 - tp0.9: 5729492.5956 - fp0.9: 139279.5152 - tn0.9: 44147595.4887 - fn0.9: 2494276.3474 - precision0.9: 0.9
  761 - recall0.9: 0.7005 - accuracy: 0.9620 - auc: 0.9747 - val_loss: 0.9282 - val_tp0.1: 3074924.0000 - val_fp0.1: 2026526.0000 - val_tn0.1: 20054118.0000 - val_fn0.1: 1058829.0000 - val_precision0.1: 0.6028 - val_recall0.1: 0.7439 - v
  al_tp0.3: 2384857.0000 - val_fp0.3: 1512374.0000 - val_tn0.3: 20568268.0000 - val_fn0.3: 1748896.0000 - val_precision0.3: 0.6119 - val_recall0.3: 0.5769 - val_tp0.5: 1874790.0000 - val_fp0.5: 1177108.0000 - val_tn0.5: 20903548.0000 - v
  al_fn0.5: 2258963.0000 - val_precision0.5: 0.6143 - val_recall0.5: 0.4535 - val_tp0.7: 1433826.0000 - val_fp0.7: 875539.0000 - val_tn0.7: 21205108.0000 - val_fn0.7: 2699927.0000 - val_precision0.7: 0.6209 - val_recall0.7: 0.3469 - val_
  tp0.9: 876571.0000 - val_fp0.9: 463067.0000 - val_tn0.9: 21617596.0000 - val_fn0.9: 3257182.0000 - val_precision0.9: 0.6543 - val_recall0.9: 0.2121 - val_accuracy: 0.8689 - val_auc: 0.8345
  Epoch 29/50
  1280/1280 [==============================] - 3301s 3s/step - loss: 0.2631 - tp0.1: 7928634.4387 - fp0.1: 3499371.4231 - tn0.1: 40746977.2912 - fn0.1: 335655.7486 - precision0.1: 0.6931 - recall0.1: 0.9590 - tp0.3: 7590846.6565 - fp0.3:
   1745234.9133 - tn0.3: 42501135.9079 - fn0.3: 673443.5308 - precision0.3: 0.8123 - recall0.3: 0.9172 - tp0.5: 7228004.8962 - fp0.5: 987598.6276 - tn0.5: 43258775.8423 - fn0.5: 1036285.2912 - precision0.5: 0.8797 - recall0.5: 0.8722 - t
  p0.7: 6751780.5855 - fp0.7: 528906.3060 - tn0.7: 43717457.8899 - fn0.7: 1512509.6019 - precision0.7: 0.9276 - recall0.7: 0.8149 - tp0.9: 5739049.2662 - fp0.9: 134220.8212 - tn0.9: 44112121.8931 - fn0.9: 2525240.9212 - precision0.9: 0.9
  773 - recall0.9: 0.6932 - accuracy: 0.9610 - auc: 0.9735 - val_loss: 0.9733 - val_tp0.1: 2953631.0000 - val_fp0.1: 2097398.0000 - val_tn0.1: 20016580.0000 - val_fn0.1: 1146781.0000 - val_precision0.1: 0.5848 - val_recall0.1: 0.7203 - v
  al_tp0.3: 2295553.0000 - val_fp0.3: 1572391.0000 - val_tn0.3: 20541608.0000 - val_fn0.3: 1804859.0000 - val_precision0.3: 0.5935 - val_recall0.3: 0.5598 - val_tp0.5: 1907607.0000 - val_fp0.5: 1337407.0000 - val_tn0.5: 20776580.0000 - v
  al_fn0.5: 2192805.0000 - val_precision0.5: 0.5879 - val_recall0.5: 0.4652 - val_tp0.7: 1521536.0000 - val_fp0.7: 1008010.0000 - val_tn0.7: 21105980.0000 - val_fn0.7: 2578876.0000 - val_precision0.7: 0.6015 - val_recall0.7: 0.3711 - val
  _tp0.9: 972957.0000 - val_fp0.9: 483731.0000 - val_tn0.9: 21630266.0000 - val_fn0.9: 3127455.0000 - val_precision0.9: 0.6679 - val_recall0.9: 0.2373 - val_accuracy: 0.8653 - val_auc: 0.8217
  Epoch 30/50
  1280/1280 [==============================] - 3328s 3s/step - loss: 0.2476 - tp0.1: 7928433.5863 - fp0.1: 3472550.0773 - tn0.1: 40794809.0422 - fn0.1: 314855.4317 - precision0.1: 0.6960 - recall0.1: 0.9627 - tp0.3: 7601842.5277 - fp0.3:
   1687439.2631 - tn0.3: 42579912.0640 - fn0.3: 641446.4902 - precision0.3: 0.8211 - recall0.3: 0.9228 - tp0.5: 7249994.1311 - fp0.5: 954111.0289 - tn0.5: 43313245.2014 - fn0.5: 993294.8868 - precision0.5: 0.8862 - recall0.5: 0.8801 - tp
  0.7: 6788340.7728 - fp0.7: 512779.7322 - tn0.7: 43754579.6230 - fn0.7: 1454948.2451 - precision0.7: 0.9317 - recall0.7: 0.8242 - tp0.9: 5804142.8977 - fp0.9: 135882.9547 - tn0.9: 44131491.4426 - fn0.9: 2439146.1202 - precision0.9: 0.97
  77 - recall0.9: 0.7071 - accuracy: 0.9638 - auc: 0.9759 - val_loss: 0.9000 - val_tp0.1: 3155596.0000 - val_fp0.1: 2364539.0000 - val_tn0.1: 19719286.0000 - val_fn0.1: 974987.0000 - val_precision0.1: 0.5717 - val_recall0.1: 0.7640 - val
  _tp0.3: 2555024.0000 - val_fp0.3: 1820537.0000 - val_tn0.3: 20263286.0000 - val_fn0.3: 1575559.0000 - val_precision0.3: 0.5839 - val_recall0.3: 0.6186 - val_tp0.5: 2171845.0000 - val_fp0.5: 1530813.0000 - val_tn0.5: 20553012.0000 - val
  _fn0.5: 1958738.0000 - val_precision0.5: 0.5866 - val_recall0.5: 0.5258 - val_tp0.7: 1696953.0000 - val_fp0.7: 1088079.0000 - val_tn0.7: 20995736.0000 - val_fn0.7: 2433630.0000 - val_precision0.7: 0.6093 - val_recall0.7: 0.4108 - val_t
  p0.9: 986911.0000 - val_fp0.9: 596529.0000 - val_tn0.9: 21487282.0000 - val_fn0.9: 3143672.0000 - val_precision0.9: 0.6233 - val_recall0.9: 0.2389 - val_accuracy: 0.8669 - val_auc: 0.8372
  Epoch 31/50
  1280/1280 [==============================] - 3327s 3s/step - loss: 0.2474 - tp0.1: 7893463.6916 - fp0.1: 3376652.6175 - tn0.1: 40920762.5230 - fn0.1: 319751.0945 - precision0.1: 0.6988 - recall0.1: 0.9618 - tp0.3: 7584152.9766 - fp0.3:
   1687559.6604 - tn0.3: 42609861.8407 - fn0.3: 629061.8095 - precision0.3: 0.8167 - recall0.3: 0.9246 - tp0.5: 7249654.8931 - fp0.5: 952218.5824 - tn0.5: 43345227.1788 - fn0.5: 963559.8931 - precision0.5: 0.8832 - recall0.5: 0.8838 - tp
  0.7: 6813905.7268 - fp0.7: 516513.1452 - tn0.7: 43780937.7900 - fn0.7: 1399309.0593 - precision0.7: 0.9292 - recall0.7: 0.8309 - tp0.9: 5825876.9290 - fp0.9: 123739.9094 - tn0.9: 44173704.9789 - fn0.9: 2387337.8571 - precision0.9: 0.97
  94 - recall0.9: 0.7105 - accuracy: 0.9638 - auc: 0.9755 - val_loss: 1.0137 - val_tp0.1: 2843857.0000 - val_fp0.1: 2045215.0000 - val_tn0.1: 20132800.0000 - val_fn0.1: 1192532.0000 - val_precision0.1: 0.5817 - val_recall0.1: 0.7046 - va
  l_tp0.3: 2207535.0000 - val_fp0.3: 1595385.0000 - val_tn0.3: 20582624.0000 - val_fn0.3: 1828854.0000 - val_precision0.3: 0.5805 - val_recall0.3: 0.5469 - val_tp0.5: 1835971.0000 - val_fp0.5: 1390695.0000 - val_tn0.5: 20787306.0000 - va
  l_fn0.5: 2200418.0000 - val_precision0.5: 0.5690 - val_recall0.5: 0.4549 - val_tp0.7: 1407052.0000 - val_fp0.7: 1033277.0000 - val_tn0.7: 21144740.0000 - val_fn0.7: 2629337.0000 - val_precision0.7: 0.5766 - val_recall0.7: 0.3486 - val_
  tp0.9: 786994.0000 - val_fp0.9: 502079.0000 - val_tn0.9: 21675932.0000 - val_fn0.9: 3249395.0000 - val_precision0.9: 0.6105 - val_recall0.9: 0.1950 - val_accuracy: 0.8630 - val_auc: 0.8121
  Epoch 32/50
  1280/1280 [==============================] - 3314s 3s/step - loss: 0.2529 - tp0.1: 7921204.5262 - fp0.1: 3486226.5160 - tn0.1: 40787447.0554 - fn0.1: 315795.7416 - precision0.1: 0.6929 - recall0.1: 0.9624 - tp0.3: 7593409.2264 - fp0.3:
   1719995.1319 - tn0.3: 42553646.5496 - fn0.3: 643591.0414 - precision0.3: 0.8139 - recall0.3: 0.9227 - tp0.5: 7240041.4294 - fp0.5: 971849.9313 - tn0.5: 43301829.9984 - fn0.5: 996958.8384 - precision0.5: 0.8807 - recall0.5: 0.8794 - tp
  0.7: 6782718.2459 - fp0.7: 527577.0203 - tn0.7: 43746081.8454 - fn0.7: 1454282.0219 - precision0.7: 0.9273 - recall0.7: 0.8235 - tp0.9: 5780993.2069 - fp0.9: 137794.4926 - tn0.9: 44135860.1952 - fn0.9: 2456007.0609 - precision0.9: 0.97
  62 - recall0.9: 0.7029 - accuracy: 0.9623 - auc: 0.9755 - val_loss: 0.9139 - val_tp0.1: 2984184.0000 - val_fp0.1: 1853144.0000 - val_tn0.1: 20302900.0000 - val_fn0.1: 1074177.0000 - val_precision0.1: 0.6169 - val_recall0.1: 0.7353 - va
  l_tp0.3: 2282389.0000 - val_fp0.3: 1253127.0000 - val_tn0.3: 20902908.0000 - val_fn0.3: 1775972.0000 - val_precision0.3: 0.6456 - val_recall0.3: 0.5624 - val_tp0.5: 1856088.0000 - val_fp0.5: 1008268.0000 - val_tn0.5: 21147768.0000 - va
  l_fn0.5: 2202273.0000 - val_precision0.5: 0.6480 - val_recall0.5: 0.4573 - val_tp0.7: 1412568.0000 - val_fp0.7: 746394.0000 - val_tn0.7: 21409650.0000 - val_fn0.7: 2645793.0000 - val_precision0.7: 0.6543 - val_recall0.7: 0.3481 - val_t
  p0.9: 859420.0000 - val_fp0.9: 382687.0000 - val_tn0.9: 21773356.0000 - val_fn0.9: 3198941.0000 - val_precision0.9: 0.6919 - val_recall0.9: 0.2118 - val_accuracy: 0.8775 - val_auc: 0.8374
  Epoch 33/50
  1280/1280 [==============================] - 3327s 3s/step - loss: 0.2503 - tp0.1: 8047970.1561 - fp0.1: 3391824.5386 - tn0.1: 40757551.3286 - fn0.1: 313307.7994 - precision0.1: 0.7045 - recall0.1: 0.9628 - tp0.3: 7729505.2756 - fp0.3:
   1696243.9040 - tn0.3: 42453121.2498 - fn0.3: 631772.6799 - precision0.3: 0.8217 - recall0.3: 0.9244 - tp0.5: 7388192.9508 - fp0.5: 982075.0718 - tn0.5: 43167288.1382 - fn0.5: 973085.0047 - precision0.5: 0.8838 - recall0.5: 0.8830 - tp
  0.7: 6931423.2888 - fp0.7: 537102.3271 - tn0.7: 43612270.6479 - fn0.7: 1429854.6667 - precision0.7: 0.9287 - recall0.7: 0.8280 - tp0.9: 5907919.1241 - fp0.9: 136823.9852 - tn0.9: 44012543.5980 - fn0.9: 2453358.8314 - precision0.9: 0.97
  77 - recall0.9: 0.7060 - accuracy: 0.9628 - auc: 0.9760 - val_loss: 1.0074 - val_tp0.1: 3175440.0000 - val_fp0.1: 2183311.0000 - val_tn0.1: 19707500.0000 - val_fn0.1: 1148150.0000 - val_precision0.1: 0.5926 - val_recall0.1: 0.7344 - va
  l_tp0.3: 2561350.0000 - val_fp0.3: 1772975.0000 - val_tn0.3: 20117834.0000 - val_fn0.3: 1762240.0000 - val_precision0.3: 0.5909 - val_recall0.3: 0.5924 - val_tp0.5: 2236272.0000 - val_fp0.5: 1567765.0000 - val_tn0.5: 20323044.0000 - va
  l_fn0.5: 2087318.0000 - val_precision0.5: 0.5879 - val_recall0.5: 0.5172 - val_tp0.7: 1860628.0000 - val_fp0.7: 1235466.0000 - val_tn0.7: 20655340.0000 - val_fn0.7: 2462962.0000 - val_precision0.7: 0.6010 - val_recall0.7: 0.4303 - val_
  tp0.9: 1296792.0000 - val_fp0.9: 704030.0000 - val_tn0.9: 21186792.0000 - val_fn0.9: 3026798.0000 - val_precision0.9: 0.6481 - val_recall0.9: 0.2999 - val_accuracy: 0.8606 - val_auc: 0.8254
  Epoch 34/50
  1280/1280 [==============================] - 3324s 3s/step - loss: 0.2507 - tp0.1: 7911580.4098 - fp0.1: 3345203.1913 - tn0.1: 40931131.7447 - fn0.1: 322742.2100 - precision0.1: 0.7028 - recall0.1: 0.9609 - tp0.3: 7599881.2022 - fp0.3:
   1647150.7923 - tn0.3: 42629169.7291 - fn0.3: 634441.4176 - precision0.3: 0.8220 - recall0.3: 0.9223 - tp0.5: 7270919.3005 - fp0.5: 951517.5379 - tn0.5: 43324815.6550 - fn0.5: 963403.3193 - precision0.5: 0.8843 - recall0.5: 0.8819 - tp
  0.7: 6834573.1491 - fp0.7: 523079.1390 - tn0.7: 43753256.2943 - fn0.7: 1399749.4707 - precision0.7: 0.9292 - recall0.7: 0.8288 - tp0.9: 5841839.3435 - fp0.9: 132340.8064 - tn0.9: 44144004.1702 - fn0.9: 2392483.2763 - precision0.9: 0.97
  80 - recall0.9: 0.7081 - accuracy: 0.9633 - auc: 0.9751 - val_loss: 0.9237 - val_tp0.1: 3263836.0000 - val_fp0.1: 1919441.0000 - val_tn0.1: 19948316.0000 - val_fn0.1: 1082810.0000 - val_precision0.1: 0.6297 - val_recall0.1: 0.7509 - va
  l_tp0.3: 2560242.0000 - val_fp0.3: 1398314.0000 - val_tn0.3: 20469446.0000 - val_fn0.3: 1786404.0000 - val_precision0.3: 0.6468 - val_recall0.3: 0.5890 - val_tp0.5: 2148456.0000 - val_fp0.5: 1146771.0000 - val_tn0.5: 20720984.0000 - va
  l_fn0.5: 2198190.0000 - val_precision0.5: 0.6520 - val_recall0.5: 0.4943 - val_tp0.7: 1663813.0000 - val_fp0.7: 806718.0000 - val_tn0.7: 21061036.0000 - val_fn0.7: 2682833.0000 - val_precision0.7: 0.6735 - val_recall0.7: 0.3828 - val_t
  p0.9: 979165.0000 - val_fp0.9: 394579.0000 - val_tn0.9: 21473184.0000 - val_fn0.9: 3367481.0000 - val_precision0.9: 0.7128 - val_recall0.9: 0.2253 - val_accuracy: 0.8724 - val_auc: 0.8405
  Epoch 35/50
  1280/1280 [==============================] - 3310s 3s/step - loss: 0.2614 - tp0.1: 7937448.7291 - fp0.1: 3421955.0578 - tn0.1: 40822462.2030 - fn0.1: 328785.1717 - precision0.1: 0.6999 - recall0.1: 0.9579 - tp0.3: 7621030.6534 - fp0.3:
   1703065.0414 - tn0.3: 42541374.6472 - fn0.3: 645203.2475 - precision0.3: 0.8174 - recall0.3: 0.9200 - tp0.5: 7279229.2404 - fp0.5: 977595.1093 - tn0.5: 43266832.8696 - fn0.5: 987004.6604 - precision0.5: 0.8809 - recall0.5: 0.8792 - tp
  0.7: 6838107.3216 - fp0.7: 536670.7908 - tn0.7: 43707710.4473 - fn0.7: 1428126.5792 - precision0.7: 0.9261 - recall0.7: 0.8262 - tp0.9: 5811902.2927 - fp0.9: 135171.2490 - tn0.9: 44109247.1507 - fn0.9: 2454331.6081 - precision0.9: 0.97
  71 - recall0.9: 0.7018 - accuracy: 0.9620 - auc: 0.9732 - val_loss: 0.8013 - val_tp0.1: 3284754.0000 - val_fp0.1: 2344554.0000 - val_tn0.1: 19771954.0000 - val_fn0.1: 813139.0000 - val_precision0.1: 0.5835 - val_recall0.1: 0.8016 - val
  _tp0.3: 2575560.0000 - val_fp0.3: 1777769.0000 - val_tn0.3: 20338744.0000 - val_fn0.3: 1522333.0000 - val_precision0.3: 0.5916 - val_recall0.3: 0.6285 - val_tp0.5: 2179640.0000 - val_fp0.5: 1496577.0000 - val_tn0.5: 20619926.0000 - val
  _fn0.5: 1918253.0000 - val_precision0.5: 0.5929 - val_recall0.5: 0.5319 - val_tp0.7: 1701082.0000 - val_fp0.7: 1053012.0000 - val_tn0.7: 21063498.0000 - val_fn0.7: 2396811.0000 - val_precision0.7: 0.6177 - val_recall0.7: 0.4151 - val_t
  p0.9: 989696.0000 - val_fp0.9: 527144.0000 - val_tn0.9: 21589364.0000 - val_fn0.9: 3108197.0000 - val_precision0.9: 0.6525 - val_recall0.9: 0.2415 - val_accuracy: 0.8697 - val_auc: 0.8572
  Epoch 36/50
  1280/1280 [==============================] - 3341s 3s/step - loss: 0.2585 - tp0.1: 7873502.4785 - fp0.1: 3351134.0312 - tn0.1: 40955766.1046 - fn0.1: 330264.3185 - precision0.1: 0.6971 - recall0.1: 0.9588 - tp0.3: 7549771.1140 - fp0.3:
   1661219.6745 - tn0.3: 42645646.7143 - fn0.3: 653995.6831 - precision0.3: 0.8162 - recall0.3: 0.9183 - tp0.5: 7211353.4215 - fp0.5: 964176.2467 - tn0.5: 43342703.0515 - fn0.5: 992413.3755 - precision0.5: 0.8793 - recall0.5: 0.8765 - tp
  0.7: 6769812.5121 - fp0.7: 536905.1553 - tn0.7: 43770008.0851 - fn0.7: 1433954.2849 - precision0.7: 0.9245 - recall0.7: 0.8228 - tp0.9: 5783984.8657 - fp0.9: 143730.5753 - tn0.9: 44163162.1390 - fn0.9: 2419781.9313 - precision0.9: 0.97
  52 - recall0.9: 0.7042 - accuracy: 0.9619 - auc: 0.9736 - val_loss: 0.7459 - val_tp0.1: 3472629.0000 - val_fp0.1: 2432774.0000 - val_tn0.1: 19591824.0000 - val_fn0.1: 717165.0000 - val_precision0.1: 0.5880 - val_recall0.1: 0.8288 - val
  _tp0.3: 2809063.0000 - val_fp0.3: 1835029.0000 - val_tn0.3: 20189584.0000 - val_fn0.3: 1380731.0000 - val_precision0.3: 0.6049 - val_recall0.3: 0.6705 - val_tp0.5: 2315959.0000 - val_fp0.5: 1490932.0000 - val_tn0.5: 20533678.0000 - val
  _fn0.5: 1873835.0000 - val_precision0.5: 0.6084 - val_recall0.5: 0.5528 - val_tp0.7: 1776363.0000 - val_fp0.7: 1094378.0000 - val_tn0.7: 20930232.0000 - val_fn0.7: 2413431.0000 - val_precision0.7: 0.6188 - val_recall0.7: 0.4240 - val_t
  p0.9: 1035383.0000 - val_fp0.9: 541763.0000 - val_tn0.9: 21482840.0000 - val_fn0.9: 3154411.0000 - val_precision0.9: 0.6565 - val_recall0.9: 0.2471 - val_accuracy: 0.8716 - val_auc: 0.8687
  Epoch 37/50
  1280/1280 [==============================] - 3295s 3s/step - loss: 0.2437 - tp0.1: 8016183.8337 - fp0.1: 3307100.4614 - tn0.1: 40879834.5652 - fn0.1: 307556.7455 - precision0.1: 0.7070 - recall0.1: 0.9629 - tp0.3: 7712574.3755 - fp0.3:
   1659818.0539 - tn0.3: 42527102.4731 - fn0.3: 611166.2037 - precision0.3: 0.8224 - recall0.3: 0.9270 - tp0.5: 7388699.9188 - fp0.5: 964567.5652 - tn0.5: 43222331.7237 - fn0.5: 935040.6604 - precision0.5: 0.8843 - recall0.5: 0.8884 - tp
  0.7: 6943361.7510 - fp0.7: 525733.5105 - tn0.7: 43661155.2553 - fn0.7: 1380378.8283 - precision0.7: 0.9294 - recall0.7: 0.8349 - tp0.9: 5919522.7244 - fp0.9: 126122.8407 - tn0.9: 44060797.3005 - fn0.9: 2404217.8548 - precision0.9: 0.97
  90 - recall0.9: 0.7122 - accuracy: 0.9641 - auc: 0.9762 - val_loss: 0.8760 - val_tp0.1: 3352442.0000 - val_fp0.1: 2392549.0000 - val_tn0.1: 19544092.0000 - val_fn0.1: 925309.0000 - val_precision0.1: 0.5835 - val_recall0.1: 0.7837 - val
  _tp0.3: 2658571.0000 - val_fp0.3: 1872790.0000 - val_tn0.3: 20063866.0000 - val_fn0.3: 1619180.0000 - val_precision0.3: 0.5867 - val_recall0.3: 0.6215 - val_tp0.5: 2313864.0000 - val_fp0.5: 1697140.0000 - val_tn0.5: 20239508.0000 - val
  _fn0.5: 1963887.0000 - val_precision0.5: 0.5769 - val_recall0.5: 0.5409 - val_tp0.7: 1882798.0000 - val_fp0.7: 1331096.0000 - val_tn0.7: 20605548.0000 - val_fn0.7: 2394953.0000 - val_precision0.7: 0.5858 - val_recall0.7: 0.4401 - val_t
  p0.9: 1104516.0000 - val_fp0.9: 586104.0000 - val_tn0.9: 21350548.0000 - val_fn0.9: 3173235.0000 - val_precision0.9: 0.6533 - val_recall0.9: 0.2582 - val_accuracy: 0.8603 - val_auc: 0.8446
  Epoch 38/50
  1280/1280 [==============================] - 3369s 3s/step - loss: 0.2413 - tp0.1: 7956859.0180 - fp0.1: 3276078.4895 - tn0.1: 40979600.7861 - fn0.1: 298124.8735 - precision0.1: 0.7090 - recall0.1: 0.9634 - tp0.3: 7646937.8517 - fp0.3:
   1590794.5269 - tn0.3: 42664873.7775 - fn0.3: 608046.0398 - precision0.3: 0.8280 - recall0.3: 0.9258 - tp0.5: 7313832.8025 - fp0.5: 911027.5652 - tn0.5: 43344624.5683 - fn0.5: 941151.0890 - precision0.5: 0.8894 - recall0.5: 0.8851 - tp
  0.7: 6881091.8657 - fp0.7: 495880.5730 - tn0.7: 43759809.8337 - fn0.7: 1373892.0258 - precision0.7: 0.9331 - recall0.7: 0.8326 - tp0.9: 5927602.7518 - fp0.9: 127784.9633 - tn0.9: 44127895.0578 - fn0.9: 2327381.1397 - precision0.9: 0.97
  90 - recall0.9: 0.7159 - accuracy: 0.9644 - auc: 0.9765 - val_loss: 4.9077 - val_tp0.1: 3486973.0000 - val_fp0.1: 2850284.0000 - val_tn0.1: 19285722.0000 - val_fn0.1: 591422.0000 - val_precision0.1: 0.5502 - val_recall0.1: 0.8550 - val
  _tp0.3: 2787062.0000 - val_fp0.3: 2180072.0000 - val_tn0.3: 19955928.0000 - val_fn0.3: 1291333.0000 - val_precision0.3: 0.5611 - val_recall0.3: 0.6834 - val_tp0.5: 2371435.0000 - val_fp0.5: 2041150.0000 - val_tn0.5: 20094856.0000 - val
  _fn0.5: 1706960.0000 - val_precision0.5: 0.5374 - val_recall0.5: 0.5815 - val_tp0.7: 2052507.0000 - val_fp0.7: 1911080.0000 - val_tn0.7: 20224928.0000 - val_fn0.7: 2025888.0000 - val_precision0.7: 0.5178 - val_recall0.7: 0.5033 - val_t
  p0.9: 1593600.0000 - val_fp0.9: 1598028.0000 - val_tn0.9: 20537976.0000 - val_fn0.9: 2484795.0000 - val_precision0.9: 0.4993 - val_recall0.9: 0.3907 - val_accuracy: 0.8570 - val_auc: 0.8622
  Epoch 39/50
  1280/1280 [==============================] - 3380s 3s/step - loss: 0.2414 - tp0.1: 7803821.4372 - fp0.1: 3354269.5839 - tn0.1: 41052380.7252 - fn0.1: 300171.4941 - precision0.1: 0.6956 - recall0.1: 0.9633 - tp0.3: 7497167.4637 - fp0.3:
   1643206.8829 - tn0.3: 42763432.2763 - fn0.3: 606825.4676 - precision0.3: 0.8198 - recall0.3: 0.9250 - tp0.5: 7161284.4465 - fp0.5: 930784.7471 - tn0.5: 43475886.7112 - fn0.5: 942708.4848 - precision0.5: 0.8867 - recall0.5: 0.8828 - tp
  0.7: 6716037.1132 - fp0.7: 498289.9742 - tn0.7: 43908382.7361 - fn0.7: 1387955.8181 - precision0.7: 0.9325 - recall0.7: 0.8275 - tp0.9: 5745482.9321 - fp0.9: 129539.1460 - tn0.9: 44277093.8798 - fn0.9: 2358509.9992 - precision0.9: 0.97
  87 - recall0.9: 0.7074 - accuracy: 0.9647 - auc: 0.9763 - val_loss: 0.8416 - val_tp0.1: 3152764.0000 - val_fp0.1: 1979016.0000 - val_tn0.1: 20133118.0000 - val_fn0.1: 949499.0000 - val_precision0.1: 0.6144 - val_recall0.1: 0.7685 - val
  _tp0.3: 2438228.0000 - val_fp0.3: 1470054.0000 - val_tn0.3: 20642072.0000 - val_fn0.3: 1664035.0000 - val_precision0.3: 0.6239 - val_recall0.3: 0.5944 - val_tp0.5: 1977683.0000 - val_fp0.5: 1217609.0000 - val_tn0.5: 20894528.0000 - val
  _fn0.5: 2124580.0000 - val_precision0.5: 0.6189 - val_recall0.5: 0.4821 - val_tp0.7: 1503343.0000 - val_fp0.7: 862897.0000 - val_tn0.7: 21249240.0000 - val_fn0.7: 2598920.0000 - val_precision0.7: 0.6353 - val_recall0.7: 0.3665 - val_tp
  0.9: 896812.0000 - val_fp0.9: 377234.0000 - val_tn0.9: 21734904.0000 - val_fn0.9: 3205451.0000 - val_precision0.9: 0.7039 - val_recall0.9: 0.2186 - val_accuracy: 0.8725 - val_auc: 0.8477
  Epoch 40/50
  1280/1280 [==============================] - 3386s 3s/step - loss: 0.2375 - tp0.1: 8019265.8907 - fp0.1: 3238880.4543 - tn0.1: 40953284.0133 - fn0.1: 299240.1600 - precision0.1: 0.7138 - recall0.1: 0.9637 - tp0.3: 7719720.4668 - fp0.3:
   1605122.1085 - tn0.3: 42587020.9703 - fn0.3: 598785.5839 - precision0.3: 0.8287 - recall0.3: 0.9280 - tp0.5: 7396785.5230 - fp0.5: 927286.7697 - tn0.5: 43264845.8103 - fn0.5: 921720.5277 - precision0.5: 0.8890 - recall0.5: 0.8895 - tp
  0.7: 6965247.9297 - fp0.7: 501804.8829 - tn0.7: 43690323.6995 - fn0.7: 1353258.1210 - precision0.7: 0.9328 - recall0.7: 0.8383 - tp0.9: 5960658.1421 - fp0.9: 123056.3981 - tn0.9: 44069097.9461 - fn0.9: 2357847.9087 - precision0.9: 0.97
  98 - recall0.9: 0.7181 - accuracy: 0.9649 - auc: 0.9769 - val_loss: 0.7336 - val_tp0.1: 3398410.0000 - val_fp0.1: 2457064.0000 - val_tn0.1: 19690942.0000 - val_fn0.1: 667979.0000 - val_precision0.1: 0.5804 - val_recall0.1: 0.8357 - val
  _tp0.3: 2749737.0000 - val_fp0.3: 1843797.0000 - val_tn0.3: 20304204.0000 - val_fn0.3: 1316652.0000 - val_precision0.3: 0.5986 - val_recall0.3: 0.6762 - val_tp0.5: 2345968.0000 - val_fp0.5: 1609916.0000 - val_tn0.5: 20538100.0000 - val
  _fn0.5: 1720421.0000 - val_precision0.5: 0.5930 - val_recall0.5: 0.5769 - val_tp0.7: 1892495.0000 - val_fp0.7: 1225411.0000 - val_tn0.7: 20922604.0000 - val_fn0.7: 2173894.0000 - val_precision0.7: 0.6070 - val_recall0.7: 0.4654 - val_t
  p0.9: 1133144.0000 - val_fp0.9: 662854.0000 - val_tn0.9: 21485156.0000 - val_fn0.9: 2933245.0000 - val_precision0.9: 0.6309 - val_recall0.9: 0.2787 - val_accuracy: 0.8730 - val_auc: 0.8721
  Epoch 41/50
  1280/1280 [==============================] - 3385s 3s/step - loss: 0.2342 - tp0.1: 7776741.3099 - fp0.1: 3289425.7650 - tn0.1: 41160217.6448 - fn0.1: 284258.4153 - precision0.1: 0.7028 - recall0.1: 0.9655 - tp0.3: 7475615.4660 - fp0.3:
   1607131.3685 - tn0.3: 42842547.8212 - fn0.3: 585384.2592 - precision0.3: 0.8230 - recall0.3: 0.9281 - tp0.5: 7146650.6237 - fp0.5: 908662.7939 - tn0.5: 43541000.6799 - fn0.5: 914349.1015 - precision0.5: 0.8871 - recall0.5: 0.8872 - tp
  0.7: 6711653.7861 - fp0.7: 487381.3060 - tn0.7: 43962279.2373 - fn0.7: 1349345.9391 - precision0.7: 0.9322 - recall0.7: 0.8330 - tp0.9: 5745993.2186 - fp0.9: 124517.9758 - tn0.9: 44325130.4988 - fn0.9: 2315006.5066 - precision0.9: 0.97
  85 - recall0.9: 0.7118 - accuracy: 0.9651 - auc: 0.9776 - val_loss: 0.9870 - val_tp0.1: 3053092.0000 - val_fp0.1: 1964937.0000 - val_tn0.1: 20015960.0000 - val_fn0.1: 1180406.0000 - val_precision0.1: 0.6084 - val_recall0.1: 0.7212 - va
  l_tp0.3: 2284036.0000 - val_fp0.3: 1477236.0000 - val_tn0.3: 20503676.0000 - val_fn0.3: 1949462.0000 - val_precision0.3: 0.6073 - val_recall0.3: 0.5395 - val_tp0.5: 1899487.0000 - val_fp0.5: 1256383.0000 - val_tn0.5: 20724524.0000 - va
  l_fn0.5: 2334011.0000 - val_precision0.5: 0.6019 - val_recall0.5: 0.4487 - val_tp0.7: 1422839.0000 - val_fp0.7: 921281.0000 - val_tn0.7: 21059624.0000 - val_fn0.7: 2810659.0000 - val_precision0.7: 0.6070 - val_recall0.7: 0.3361 - val_t
  p0.9: 829426.0000 - val_fp0.9: 481727.0000 - val_tn0.9: 21499180.0000 - val_fn0.9: 3404072.0000 - val_precision0.9: 0.6326 - val_recall0.9: 0.1959 - val_accuracy: 0.8630 - val_auc: 0.8226
  Epoch 42/50
  1280/1280 [==============================] - 3315s 3s/step - loss: 0.2346 - tp0.1: 8105380.3060 - fp0.1: 3217998.5480 - tn0.1: 40897704.3310 - fn0.1: 289593.6690 - precision0.1: 0.7181 - recall0.1: 0.9656 - tp0.3: 7802382.4910 - fp0.3:
   1604270.8267 - tn0.3: 42511413.2233 - fn0.3: 592591.4840 - precision0.3: 0.8318 - recall0.3: 0.9298 - tp0.5: 7475287.1304 - fp0.5: 928889.2771 - tn0.5: 43186785.4551 - fn0.5: 919686.8447 - precision0.5: 0.8906 - recall0.5: 0.8917 - tp
  0.7: 7046455.8064 - fp0.7: 507457.0898 - tn0.7: 43608208.9945 - fn0.7: 1348518.1686 - precision0.7: 0.9334 - recall0.7: 0.8413 - tp0.9: 6058878.7190 - fp0.9: 128713.2311 - tn0.9: 43986958.0468 - fn0.9: 2336095.2560 - precision0.9: 0.97
  95 - recall0.9: 0.7240 - accuracy: 0.9650 - auc: 0.9779 - val_loss: 0.7327 - val_tp0.1: 3402300.0000 - val_fp0.1: 2653288.0000 - val_tn0.1: 19553712.0000 - val_fn0.1: 605094.0000 - val_precision0.1: 0.5618 - val_recall0.1: 0.8490 - val
  _tp0.3: 2812815.0000 - val_fp0.3: 2101002.0000 - val_tn0.3: 20106008.0000 - val_fn0.3: 1194579.0000 - val_precision0.3: 0.5724 - val_recall0.3: 0.7019 - val_tp0.5: 2455006.0000 - val_fp0.5: 1878473.0000 - val_tn0.5: 20328530.0000 - val
  _fn0.5: 1552388.0000 - val_precision0.5: 0.5665 - val_recall0.5: 0.6126 - val_tp0.7: 2074881.0000 - val_fp0.7: 1530207.0000 - val_tn0.7: 20676798.0000 - val_fn0.7: 1932513.0000 - val_precision0.7: 0.5755 - val_recall0.7: 0.5178 - val_t
  p0.9: 1484583.0000 - val_fp0.9: 913417.0000 - val_tn0.9: 21293592.0000 - val_fn0.9: 2522811.0000 - val_precision0.9: 0.6191 - val_recall0.9: 0.3705 - val_accuracy: 0.8691 - val_auc: 0.8751
  Epoch 43/50
  1280/1280 [==============================] - 3323s 3s/step - loss: 0.2312 - tp0.1: 7991710.0656 - fp0.1: 3249384.0929 - tn0.1: 40992411.0070 - fn0.1: 277145.7760 - precision0.1: 0.7097 - recall0.1: 0.9664 - tp0.3: 7693901.7158 - fp0.3:
   1595395.2670 - tn0.3: 42646409.5269 - fn0.3: 574954.1257 - precision0.3: 0.8268 - recall0.3: 0.9305 - tp0.5: 7373192.4247 - fp0.5: 917265.3872 - tn0.5: 43324540.4536 - fn0.5: 895663.4169 - precision0.5: 0.8882 - recall0.5: 0.8913 - tp
  0.7: 6950466.7244 - fp0.7: 497989.1148 - tn0.7: 43743813.0945 - fn0.7: 1318389.1171 - precision0.7: 0.9326 - recall0.7: 0.8395 - tp0.9: 5981119.6440 - fp0.9: 128545.2209 - tn0.9: 44113239.2467 - fn0.9: 2287736.1975 - precision0.9: 0.97
  92 - recall0.9: 0.7218 - accuracy: 0.9653 - auc: 0.9783 - val_loss: 0.9647 - val_tp0.1: 2876343.0000 - val_fp0.1: 1660425.0000 - val_tn0.1: 20507034.0000 - val_fn0.1: 1170598.0000 - val_precision0.1: 0.6340 - val_recall0.1: 0.7107 - va
  l_tp0.3: 2209087.0000 - val_fp0.3: 1251945.0000 - val_tn0.3: 20915512.0000 - val_fn0.3: 1837854.0000 - val_precision0.3: 0.6383 - val_recall0.3: 0.5459 - val_tp0.5: 1778622.0000 - val_fp0.5: 995657.0000 - val_tn0.5: 21171812.0000 - val
  _fn0.5: 2268319.0000 - val_precision0.5: 0.6411 - val_recall0.5: 0.4395 - val_tp0.7: 1307626.0000 - val_fp0.7: 702184.0000 - val_tn0.7: 21465282.0000 - val_fn0.7: 2739315.0000 - val_precision0.7: 0.6506 - val_recall0.7: 0.3231 - val_tp
  0.9: 721015.0000 - val_fp0.9: 304685.0000 - val_tn0.9: 21862786.0000 - val_fn0.9: 3325926.0000 - val_precision0.9: 0.7029 - val_recall0.9: 0.1782 - val_accuracy: 0.8755 - val_auc: 0.8252
  Epoch 44/50
  1280/1280 [==============================] - 3324s 3s/step - loss: 0.2316 - tp0.1: 7993553.5519 - fp0.1: 3116269.6425 - tn0.1: 41108883.5878 - fn0.1: 291939.1897 - precision0.1: 0.7205 - recall0.1: 0.9644 - tp0.3: 7697513.5496 - fp0.3:
   1532368.8548 - tn0.3: 42692802.8033 - fn0.3: 587979.1920 - precision0.3: 0.8347 - recall0.3: 0.9284 - tp0.5: 7391393.2475 - fp0.5: 891003.0500 - tn0.5: 43334168.9118 - fn0.5: 894099.4941 - precision0.5: 0.8927 - recall0.5: 0.8916 - tp
  0.7: 6972420.4403 - fp0.7: 485629.1991 - tn0.7: 43739548.5152 - fn0.7: 1313072.3013 - precision0.7: 0.9351 - recall0.7: 0.8406 - tp0.9: 6005577.5371 - fp0.9: 125956.1194 - tn0.9: 44099185.2194 - fn0.9: 2279915.2045 - precision0.9: 0.97
  93 - recall0.9: 0.7244 - accuracy: 0.9660 - auc: 0.9775 - val_loss: 0.7395 - val_tp0.1: 3516454.0000 - val_fp0.1: 2931698.0000 - val_tn0.1: 19145124.0000 - val_fn0.1: 621132.0000 - val_precision0.1: 0.5453 - val_recall0.1: 0.8499 - val
  _tp0.3: 2802624.0000 - val_fp0.3: 2096050.0000 - val_tn0.3: 19980764.0000 - val_fn0.3: 1334962.0000 - val_precision0.3: 0.5721 - val_recall0.3: 0.6774 - val_tp0.5: 2395920.0000 - val_fp0.5: 1817727.0000 - val_tn0.5: 20259092.0000 - val
  _fn0.5: 1741666.0000 - val_precision0.5: 0.5686 - val_recall0.5: 0.5791 - val_tp0.7: 1884999.0000 - val_fp0.7: 1378065.0000 - val_tn0.7: 20698756.0000 - val_fn0.7: 2252587.0000 - val_precision0.7: 0.5777 - val_recall0.7: 0.4556 - val_t
  p0.9: 983346.0000 - val_fp0.9: 679335.0000 - val_tn0.9: 21397474.0000 - val_fn0.9: 3154240.0000 - val_precision0.9: 0.5914 - val_recall0.9: 0.2377 - val_accuracy: 0.8642 - val_auc: 0.8683
  Epoch 45/50
  1280/1280 [==============================] - 3312s 3s/step - loss: 0.2330 - tp0.1: 7970480.2670 - fp0.1: 3209589.4871 - tn0.1: 41051845.7307 - fn0.1: 278741.1163 - precision0.1: 0.7127 - recall0.1: 0.9666 - tp0.3: 7675031.2553 - fp0.3:
   1589895.0429 - tn0.3: 42671527.8275 - fn0.3: 574190.1280 - precision0.3: 0.8279 - recall0.3: 0.9311 - tp0.5: 7351503.7557 - fp0.5: 918595.6940 - tn0.5: 43342860.1577 - fn0.5: 897717.6276 - precision0.5: 0.8886 - recall0.5: 0.8921 - tp
  0.7: 6925182.0765 - fp0.7: 503433.8673 - tn0.7: 43758011.1593 - fn0.7: 1324039.3068 - precision0.7: 0.9318 - recall0.7: 0.8413 - tp0.9: 5948949.5558 - fp0.9: 139165.2740 - tn0.9: 44122272.1593 - fn0.9: 2300271.8275 - precision0.9: 0.97
  68 - recall0.9: 0.7246 - accuracy: 0.9651 - auc: 0.9781 - val_loss: 0.7183 - val_tp0.1: 3462944.0000 - val_fp0.1: 2547992.0000 - val_tn0.1: 19584976.0000 - val_fn0.1: 618487.0000 - val_precision0.1: 0.5761 - val_recall0.1: 0.8485 - val
  _tp0.3: 2757155.0000 - val_fp0.3: 1916541.0000 - val_tn0.3: 20216436.0000 - val_fn0.3: 1324276.0000 - val_precision0.3: 0.5899 - val_recall0.3: 0.6755 - val_tp0.5: 2406562.0000 - val_fp0.5: 1766897.0000 - val_tn0.5: 20366082.0000 - val
  _fn0.5: 1674869.0000 - val_precision0.5: 0.5766 - val_recall0.5: 0.5896 - val_tp0.7: 2083375.0000 - val_fp0.7: 1557338.0000 - val_tn0.7: 20575626.0000 - val_fn0.7: 1998056.0000 - val_precision0.7: 0.5722 - val_recall0.7: 0.5105 - val_t
  p0.9: 1330037.0000 - val_fp0.9: 696335.0000 - val_tn0.9: 21436634.0000 - val_fn0.9: 2751394.0000 - val_precision0.9: 0.6564 - val_recall0.9: 0.3259 - val_accuracy: 0.8687 - val_auc: 0.8762
  Epoch 46/50
  1280/1280 [==============================] - 3307s 3s/step - loss: 0.2304 - tp0.1: 7916214.3216 - fp0.1: 3181145.9766 - tn0.1: 41127786.5941 - fn0.1: 285496.9227 - precision0.1: 0.7125 - recall0.1: 0.9651 - tp0.3: 7621058.9899 - fp0.3:
   1584443.6526 - tn0.3: 42724483.9516 - fn0.3: 580652.2545 - precision0.3: 0.8279 - recall0.3: 0.9291 - tp0.5: 7291826.7104 - fp0.5: 896453.6901 - tn0.5: 43412477.0523 - fn0.5: 909884.5340 - precision0.5: 0.8906 - recall0.5: 0.8889 - tp
  0.7: 6852507.8103 - fp0.7: 474238.7486 - tn0.7: 43834734.6058 - fn0.7: 1349203.4340 - precision0.7: 0.9355 - recall0.7: 0.8352 - tp0.9: 5907297.4052 - fp0.9: 120870.0991 - tn0.9: 44188071.7377 - fn0.9: 2294413.8392 - precision0.9: 0.98
  03 - recall0.9: 0.7195 - accuracy: 0.9658 - auc: 0.9776 - val_loss: 0.8308 - val_tp0.1: 2925415.0000 - val_fp0.1: 999631.0000 - val_tn0.1: 21181596.0000 - val_fn0.1: 1107748.0000 - val_precision0.1: 0.7453 - val_recall0.1: 0.7253 - val
  _tp0.3: 2129554.0000 - val_fp0.3: 432959.0000 - val_tn0.3: 21748276.0000 - val_fn0.3: 1903609.0000 - val_precision0.3: 0.8310 - val_recall0.3: 0.5280 - val_tp0.5: 1616477.0000 - val_fp0.5: 255434.0000 - val_tn0.5: 21925804.0000 - val_f
  n0.5: 2416686.0000 - val_precision0.5: 0.8635 - val_recall0.5: 0.4008 - val_tp0.7: 1160120.0000 - val_fp0.7: 159480.0000 - val_tn0.7: 22021744.0000 - val_fn0.7: 2873043.0000 - val_precision0.7: 0.8791 - val_recall0.7: 0.2876 - val_tp0.
  9: 660411.0000 - val_fp0.9: 58454.0000 - val_tn0.9: 22122774.0000 - val_fn0.9: 3372752.0000 - val_precision0.9: 0.9187 - val_recall0.9: 0.1637 - val_accuracy: 0.8981 - val_auc: 0.8530
  Epoch 47/50
  1280/1280 [==============================] - 3373s 3s/step - loss: 0.2224 - tp0.1: 7905453.9578 - fp0.1: 3088525.5402 - tn0.1: 41239482.1889 - fn0.1: 277215.0070 - precision0.1: 0.7202 - recall0.1: 0.9667 - tp0.3: 7625498.5870 - fp0.3:
   1552664.1967 - tn0.3: 42775315.1663 - fn0.3: 557170.3778 - precision0.3: 0.8312 - recall0.3: 0.9328 - tp0.5: 7314136.2771 - fp0.5: 889383.1952 - tn0.5: 43438616.4731 - fn0.5: 868532.6877 - precision0.5: 0.8918 - recall0.5: 0.8950 - tp
  0.7: 6893069.8415 - fp0.7: 475384.9407 - tn0.7: 43852616.7221 - fn0.7: 1289599.1233 - precision0.7: 0.9360 - recall0.7: 0.8437 - tp0.9: 5954104.8665 - fp0.9: 121730.6745 - tn0.9: 44206239.8525 - fn0.9: 2228564.0984 - precision0.9: 0.98
  04 - recall0.9: 0.7290 - accuracy: 0.9667 - auc: 0.9788 - val_loss: 1.0397 - val_tp0.1: 2921669.0000 - val_fp0.1: 1566454.0000 - val_tn0.1: 20378636.0000 - val_fn0.1: 1347646.0000 - val_precision0.1: 0.6510 - val_recall0.1: 0.6843 - va
  l_tp0.3: 2142218.0000 - val_fp0.3: 1055434.0000 - val_tn0.3: 20889652.0000 - val_fn0.3: 2127097.0000 - val_precision0.3: 0.6699 - val_recall0.3: 0.5018 - val_tp0.5: 1695622.0000 - val_fp0.5: 805323.0000 - val_tn0.5: 21139764.0000 - val
  _fn0.5: 2573693.0000 - val_precision0.5: 0.6780 - val_recall0.5: 0.3972 - val_tp0.7: 1240483.0000 - val_fp0.7: 547415.0000 - val_tn0.7: 21397672.0000 - val_fn0.7: 3028832.0000 - val_precision0.7: 0.6938 - val_recall0.7: 0.2906 - val_tp
  0.9: 699052.0000 - val_fp0.9: 209226.0000 - val_tn0.9: 21735862.0000 - val_fn0.9: 3570263.0000 - val_precision0.9: 0.7696 - val_recall0.9: 0.1637 - val_accuracy: 0.8711 - val_auc: 0.8155
  Epoch 48/50
  1280/1280 [==============================] - 3304s 3s/step - loss: 0.2335 - tp0.1: 7940859.5363 - fp0.1: 3183249.2857 - tn0.1: 41099674.0820 - fn0.1: 286865.7822 - precision0.1: 0.7138 - recall0.1: 0.9650 - tp0.3: 7641716.9641 - fp0.3:
   1582103.0671 - tn0.3: 42700833.3349 - fn0.3: 586008.3544 - precision0.3: 0.8288 - recall0.3: 0.9284 - tp0.5: 7316574.1686 - fp0.5: 906813.2022 - tn0.5: 43376114.1671 - fn0.5: 911151.1499 - precision0.5: 0.8900 - recall0.5: 0.8886 - tp
  0.7: 6893099.7447 - fp0.7: 495988.3950 - tn0.7: 43786957.1632 - fn0.7: 1334625.5738 - precision0.7: 0.9332 - recall0.7: 0.8372 - tp0.9: 5936967.9602 - fp0.9: 132958.7853 - tn0.9: 44149966.0359 - fn0.9: 2290757.3583 - precision0.9: 0.97
  82 - recall0.9: 0.7212 - accuracy: 0.9651 - auc: 0.9775 - val_loss: 0.8885 - val_tp0.1: 3090482.0000 - val_fp0.1: 1927063.0000 - val_tn0.1: 20147076.0000 - val_fn0.1: 1049776.0000 - val_precision0.1: 0.6159 - val_recall0.1: 0.7464 - va
  l_tp0.3: 2402659.0000 - val_fp0.3: 1358720.0000 - val_tn0.3: 20715420.0000 - val_fn0.3: 1737599.0000 - val_precision0.3: 0.6388 - val_recall0.3: 0.5803 - val_tp0.5: 2026443.0000 - val_fp0.5: 1121590.0000 - val_tn0.5: 20952544.0000 - va
  l_fn0.5: 2113815.0000 - val_precision0.5: 0.6437 - val_recall0.5: 0.4894 - val_tp0.7: 1587666.0000 - val_fp0.7: 757043.0000 - val_tn0.7: 21317100.0000 - val_fn0.7: 2552592.0000 - val_precision0.7: 0.6771 - val_recall0.7: 0.3835 - val_t
  p0.9: 946633.0000 - val_fp0.9: 302967.0000 - val_tn0.9: 21771180.0000 - val_fn0.9: 3193625.0000 - val_precision0.9: 0.7575 - val_recall0.9: 0.2286 - val_accuracy: 0.8766 - val_auc: 0.8420
  Epoch 49/50
  1280/1280 [==============================] - 3305s 3s/step - loss: 0.2225 - tp0.1: 7979679.4473 - fp0.1: 3114299.9914 - tn0.1: 41145497.0710 - fn0.1: 271189.6027 - precision0.1: 0.7186 - recall0.1: 0.9674 - tp0.3: 7686659.5777 - fp0.3:
   1542401.8501 - tn0.3: 42717409.8704 - fn0.3: 564209.4723 - precision0.3: 0.8334 - recall0.3: 0.9319 - tp0.5: 7371634.7557 - fp0.5: 895979.9727 - tn0.5: 43363790.1265 - fn0.5: 879234.2943 - precision0.5: 0.8923 - recall0.5: 0.8937 - tp
  0.7: 6947267.2155 - fp0.7: 481955.5433 - tn0.7: 43777844.2342 - fn0.7: 1303601.8345 - precision0.7: 0.9356 - recall0.7: 0.8420 - tp0.9: 6003926.6534 - fp0.9: 121081.5886 - tn0.9: 44138712.2896 - fn0.9: 2246942.3966 - precision0.9: 0.98
  05 - recall0.9: 0.7264 - accuracy: 0.9665 - auc: 0.9792 - val_loss: 0.9566 - val_tp0.1: 2861835.0000 - val_fp0.1: 1548477.0000 - val_tn0.1: 20587614.0000 - val_fn0.1: 1216470.0000 - val_precision0.1: 0.6489 - val_recall0.1: 0.7017 - va
  l_tp0.3: 2113828.0000 - val_fp0.3: 1067547.0000 - val_tn0.3: 21068548.0000 - val_fn0.3: 1964477.0000 - val_precision0.3: 0.6644 - val_recall0.3: 0.5183 - val_tp0.5: 1718632.0000 - val_fp0.5: 841787.0000 - val_tn0.5: 21294304.0000 - val
  _fn0.5: 2359673.0000 - val_precision0.5: 0.6712 - val_recall0.5: 0.4214 - val_tp0.7: 1302909.0000 - val_fp0.7: 555088.0000 - val_tn0.7: 21581000.0000 - val_fn0.7: 2775396.0000 - val_precision0.7: 0.7012 - val_recall0.7: 0.3195 - val_tp
  0.9: 773038.0000 - val_fp0.9: 218091.0000 - val_tn0.9: 21918012.0000 - val_fn0.9: 3305267.0000 - val_precision0.9: 0.7800 - val_recall0.9: 0.1895 - val_accuracy: 0.8779 - val_auc: 0.8265
  Epoch 50/50
  1280/1280 [==============================] - 3294s 3s/step - loss: 0.2301 - tp0.1: 8047942.4426 - fp0.1: 3082506.1288 - tn0.1: 41101922.8275 - fn0.1: 278310.9493 - precision0.1: 0.7219 - recall0.1: 0.9663 - tp0.3: 7760370.9899 - fp0.3:
   1521186.0101 - tn0.3: 42663221.2100 - fn0.3: 565882.4020 - precision0.3: 0.8352 - recall0.3: 0.9315 - tp0.5: 7453452.6776 - fp0.5: 892538.5199 - tn0.5: 43291840.9251 - fn0.5: 872800.7143 - precision0.5: 0.8922 - recall0.5: 0.8941 - tp
  0.7: 7032066.8470 - fp0.7: 486059.3411 - tn0.7: 43698327.0984 - fn0.7: 1294186.5449 - precision0.7: 0.9348 - recall0.7: 0.8430 - tp0.9: 6085441.6674 - fp0.9: 123868.2233 - tn0.9: 44060549.1827 - fn0.9: 2240811.7244 - precision0.9: 0.98
  05 - recall0.9: 0.7278 - accuracy: 0.9656 - auc: 0.9784 - val_loss: 0.7966 - val_tp0.1: 3023429.0000 - val_fp0.1: 1958644.0000 - val_tn0.1: 20321072.0000 - val_fn0.1: 911248.0000 - val_precision0.1: 0.6069 - val_recall0.1: 0.7684 - val
  _tp0.3: 2293385.0000 - val_fp0.3: 1327982.0000 - val_tn0.3: 20951742.0000 - val_fn0.3: 1641292.0000 - val_precision0.3: 0.6333 - val_recall0.3: 0.5829 - val_tp0.5: 1893004.0000 - val_fp0.5: 1112093.0000 - val_tn0.5: 21167626.0000 - val
  _fn0.5: 2041673.0000 - val_precision0.5: 0.6299 - val_recall0.5: 0.4811 - val_tp0.7: 1460348.0000 - val_fp0.7: 748911.0000 - val_tn0.7: 21530808.0000 - val_fn0.7: 2474329.0000 - val_precision0.7: 0.6610 - val_recall0.7: 0.3711 - val_tp
  0.9: 871307.0000 - val_fp0.9: 277884.0000 - val_tn0.9: 22001836.0000 - val_fn0.9: 3063370.0000 - val_precision0.9: 0.7582 - val_recall0.9: 0.2214 - val_accuracy: 0.8797 - val_auc: 0.8508
  600/600 [==============================] - 317s 527ms/step - loss: 0.9377 - tp0.1: 5542504.0000 - fp0.1: 4670897.0000 - tn0.1: 36924880.0000 - fn0.1: 2013729.0000 - precision0.1: 0.5427 - recall0.1: 0.7335 - tp0.3: 4216657.0000 - fp0.3
  : 3293157.0000 - tn0.3: 38302600.0000 - fn0.3: 3339576.0000 - precision0.3: 0.5615 - recall0.3: 0.5580 - tp0.5: 3498904.0000 - fp0.5: 2765631.0000 - tn0.5: 38830152.0000 - fn0.5: 4057329.0000 - precision0.5: 0.5585 - recall0.5: 0.4630
  - tp0.7: 2661409.0000 - fp0.7: 1815115.0000 - tn0.7: 39780628.0000 - fn0.7: 4894824.0000 - precision0.7: 0.5945 - recall0.7: 0.3522 - tp0.9: 1573540.0000 - fp0.9: 592773.0000 - tn0.9: 41002980.0000 - fn0.9: 5982693.0000 - precision0.9:
   0.7264 - recall0.9: 0.2082 - accuracy: 0.8612 - auc: 0.8217
  2021/03/18 20:42:08 INFO mlflow.projects: === Run (ID '3cec3f26ed2d4004978c4ec37c00fba0') succeeded ===
  (tf-nightly) [ye53nis@node117 drmed-git]$
  

• test data is not saved out automatically, but can be copied from the log above

  600/600 [==============================] - 317s 527ms/step - loss: 0.9377

  …	0.1	0.3	0.5	0.7	0.9
  tp	5542504.0000	4216657.0000	3498904.0000	2661409.0000	1573540.0000
  fp	4670897.0000	3293157.0000	2765631.0000	1815115.0000	592773.0000
  fn	2013729.0000	3339576.0000	4057329.0000	4894824.0000	5982693.0000
  tn	36924880.0000	38302600.0000	38830152.0000	39780628.0000	41002980.0000
  all	49152010	49151990	49152016	49151976	49151986
  precision	0.5427	0.5615	0.5585	0.5945	0.7264
  recall	0.7335	0.5580	0.4630	0.3522	0.2082

  accuracy: 0.8612 auc: 0.8217

  0.1	actual positive	actual negative
  pred positive	0.11276251	0.095029623	Prec: 0.5427
  pred negative	0.040969413	0.75123845
  	Recall: 0.7335
  			F1: 0.62383709

  0.3	actual positive	actual negative
  pred positive	0.085788124	0.066999464	Prec: 0.5615
  pred negative	0.067943861	0.77926855
  	Recall: 0.5580
  			F1: 0.55974453

  0.5	actual positive	actual negative
  pred positive	0.071185361	0.056266888	Prec: 0.5585
  pred negative	0.082546543	0.79000121
  	Recall: 0.4630
  			F1: 0.50628585

  0.7	actual positive	actual negative
  pred positive	0.054146531	0.036928627	Prec: 0.5945
  pred negative	0.099585498	0.80933934
  	Recall: 0.3522
  			F1: 0.44234266

  0.9	actual positive	actual negative
  pred positive	0.032013762	0.012060001	Prec: 0.7264
  pred negative	0.12171824	0.83420800
  	Recall: 0.2082
  			F1: 0.32363895

• looks like pred_thresh=0.1 leads to the best results (Best F1 score)

sadly, the random sample traces were not very useful, since the spikes were quite small…

1. Git log, because some more code changes were made

   !git log -5
   

   commit 6586965ec900669bd641d69f85b4999050122502
   Author: Apoplex <oligolex@vivaldi.net>
   Date:   Tue Mar 23 20:26:11 2021 +0100
   
       Fix management of correlation failing 2
   
   commit ccdb3fb0887cce929987ef04408133917c02ce58
   Author: Apoplex <oligolex@vivaldi.net>
   Date:   Tue Mar 23 17:57:39 2021 +0100
   
       Merge plot_simulations and analyze_simulations 2
   
   commit 7951d725c9d53bbffd18c3fca8ccad05d528a368
   Author: Apoplex <oligolex@vivaldi.net>
   Date:   Sun Mar 21 12:49:47 2021 +0100
   
       Fix management of correlation failing; return nan
   
   commit 6c2146df58ba0af6a148745e7fafc7ff01a2e1b4
   Author: Apoplex <oligolex@vivaldi.net>
   Date:   Sun Mar 21 10:43:04 2021 +0100
   
       New management of correlation failing; return nan
   
   commit 940df93d2dacdb9c1951b6d8a8aa92f2c6043695
   Author: Apoplex <oligolex@vivaldi.net>
   Date:   Sat Mar 20 19:54:01 2021 +0100
   
       Merge plot_simulations and analyze_simulations
   

   %cd /beegfs/ye53nis/drmed-git
   

   /beegfs/ye53nis/drmed-git
   

   from pathlib import Path
   
   import sys
   import mlflow
   import matplotlib.pyplot as plt
   import numpy as np
   import pandas as pd
   import seaborn as sns
   mlflow.version.VERSION
   

   1.13.1
   

   sys.path.append('src/')
   from fluotracify.simulations import (
      import_simulation_from_csv as isfc,
      analyze_simulations as ans,
   )
   from fluotracify.training import build_model as bm, preprocess_data as ppd
   from fluotracify.applications import correlate, plots, correction
   from fluotracify.imports import ptu_utils as ptu
   

   import importlib
   importlib.reload(correction)
   

   <module 'fluotracify.applications.correction' from 'src/fluotracify/applications/correction.py'>
   

   folder = '/beegfs/ye53nis/saves/firstartifact_Nov2020_test'
   col_per_example = 3
   lab_thresh = 0.04
   pred_thresh = 0.1
   xunit = 1
   artifact = 0
   model_type = 1
   fwhm = 250
   run_id = '3cec3f26ed2d4004978c4ec37c00fba0'
   length_delimiter = 2**14
   

2. now load the trained model

   mlflow.set_tracking_uri('file:///beegfs/ye53nis/drmed-git/data/mlruns')
   client = mlflow.tracking.MlflowClient(tracking_uri=mlflow.get_tracking_uri())
   model_path = client.download_artifacts(run_id=run_id,
                                          path='model')
   model_keras = mlflow.keras.load_model(model_uri=model_path,
                                         custom_objects={'binary_ce_dice':bm.binary_ce_dice_loss()})
   print(model_path, '\n', model_keras)
   

   /beegfs/ye53nis/drmed-git/3cec3f26ed2d4004978c4ec37c00fba0/artifacts/model
    <tensorflow.python.keras.engine.functional.Functional object at 0x2acdc9893190>
   

• test and train data were separated beforehand. Now, I only load test data.

  folder = '/beegfs/ye53nis/saves/firstartifact_Nov2020_test'
  dataset, _, nsamples, experiment_params = isfc.import_from_csv(
            folder=folder,
            header=12,
            frac_train=1,
            col_per_example=col_per_example,
            dropindex=None,
            dropcolumns=None)
  experiment_params
  

  train 0 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.069_set010.csv
  train 1 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D50_set002.csv
  train 2 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.4_set005.csv
  train 3 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.2_set005.csv
  train 4 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D3.0_set007.csv
  train 5 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D3.0_set002.csv
  train 6 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D50_set001.csv
  train 7 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.2_set007.csv
  train 8 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.6_set009.csv
  train 9 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D10_set002.csvtrain 10 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.08_set005.csv
  train 11 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.6_set008.csv
  train 12 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.1_set005.csv
  train 13 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.4_set008.csv
  train 14 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D10_set005.csv
  train 15 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D1.0_set006.csv
  train 16 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.069_set005.csv
  train 17 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D50_set003.csv
  train 18 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.1_set001.csv
  train 19 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.08_set003.csv
  train 20 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D1.0_set003.csv
  train 21 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D1.0_set005.csv
  train 22 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.2_set002.csv
  train 23 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.1_set002.csv
  train 24 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D3.0_set004.csv
  train 25 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.08_set001.csv
  train 26 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.069_set001.csv
  train 27 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D10_set001.csv
  train 28 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.6_set003.csv
  train 29 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.4_set001.csv
  

  diffrates = experiment_params.loc['diffusion rate of molecules in micrometer^2 / s'].astype(np.float32)
  nmols = experiment_params.loc['number of fast molecules'].astype(np.float32)
  clusters = experiment_params.loc['diffusion rate of clusters in micrometer^2 / s'].astype(np.float32)
  
  dataset_sep = isfc.separate_data_and_labels(array=dataset,
                                              nsamples=nsamples,
                                              col_per_example=col_per_example)
  
  features = dataset_sep['0']
  labels_artifact = dataset_sep['1']
  labels_artifact_bool = labels_artifact > lab_thresh
  labels_puretrace = dataset_sep['2']
  

  The given DataFrame was split into 3 parts with shapes: [(16384, 3000), (16384, 3000), (16384, 3000)]
  

• Let's correct the traces with the new model and correlate them!

  corr_out = ans.correlate_simulations_corrected_by_prediction(
      model=model_keras,
      lab_thresh=lab_thresh,
      pred_thresh=pred_thresh,
      artifact=artifact,
      model_type=model_type,
      experiment_params=experiment_params,
      nsamples=nsamples,
      features=features,
      labels_artifact=labels_artifact,
      labels_puretrace=labels_puretrace,
      save_as_csv=True)
  corr_out
  

  processed correlation of 3000 traces with correction by label
  processed correlation of 3000 traces with correction by prediction
  processed correlation of 3000 traces without correction
  processed correlation of pure 3000 traces
  

  	Simulated \(D\)	Simulated \(D\_{{clust}}\)	nmol	\(D\) in \(\frac{{\mu m\^2}}{{s}}\)	\(\tau\_{{D}}\) in \(ms\)	Trace lengths	Traces used
  0	0.069	1.0	679.0	1.037836	10.860146	16384	corrupted without correction
  1	0.069	1.0	679.0	0.81403	13.846002	16384	corrupted without correction
  2	0.069	1.0	679.0	1.053716	10.696482	16384	corrupted without correction
  3	0.069	1.0	679.0	0.897022	12.564974	16384	corrupted without correction
  4	0.069	1.0	679.0	1.083577	10.401707	16384	corrupted without correction
  …	…	…	…	…	…	…	…
  11995	0.4	0.1	1939.0	0.658104	17.126562	11187	corrected by prediction
  11996	0.4	0.1	1939.0	0.336804	33.464773	15720	corrected by prediction
  11997	0.4	0.1	1939.0	0.456482	24.691128	15292	corrected by prediction
  11998	0.4	0.1	1939.0	0.411872	27.365437	12516	corrected by prediction
  11999	0.4	0.1	1939.0	0.244884	46.026059	12986	corrected by prediction

  12000 rows × 7 columns

  corr_out = pd.read_csv(filepath_or_buffer='data/exp-210204-unet/2021-03-21_correlations.csv')
  corr_out[corr_out['$D$ in $\\frac{{\mu m^2}}{{s}}$'].isna()]
  

  	Simulated \(D\)	Simulated \(D\_{{clust}}\)	nmol	\(D\) in \(\frac{{\mu m\^2}}{{s}}\)	\(\tau\_{{D}}\) in \(ms\)	Trace lengths	Traces used
  9109	50.0	0.01	2323.0	NaN	NaN	1	corrected by prediction
  9118	50.0	0.01	2323.0	NaN	NaN	6	corrected by prediction
  9131	50.0	0.01	2323.0	NaN	NaN	26	corrected by prediction
  9134	50.0	0.01	2323.0	NaN	NaN	3	corrected by prediction
  9141	50.0	0.01	2323.0	NaN	NaN	6	corrected by prediction
  9144	50.0	0.01	2323.0	NaN	NaN	17	corrected by prediction
  9163	50.0	0.01	2323.0	NaN	NaN	24	corrected by prediction
  9170	50.0	0.01	2323.0	NaN	NaN	27	corrected by prediction
  9171	50.0	0.01	2323.0	NaN	NaN	30	corrected by prediction
  9174	50.0	0.01	2323.0	NaN	NaN	8	corrected by prediction
  9183	50.0	0.01	2323.0	NaN	NaN	13	corrected by prediction
  9187	50.0	0.01	2323.0	NaN	NaN	29	corrected by prediction
  9199	50.0	0.01	2323.0	NaN	NaN	18	corrected by prediction
  9909	10.0	0.01	1396.0	NaN	NaN	24	corrected by prediction
  9922	10.0	0.01	1396.0	NaN	NaN	7	corrected by prediction
  9931	10.0	0.01	1396.0	NaN	NaN	16	corrected by prediction
  9978	10.0	0.01	1396.0	NaN	NaN	31	corrected by prediction
  9998	10.0	0.01	1396.0	NaN	NaN	9	corrected by prediction

• We loaded the correlation results from the saved out .csv file and looked at where it failed. All the cases where basically all the trace was (wrongfully) removed, was at a very slow simulated cluster speed (0.01) and a rather high simulated speed of the molecules we are interested in (10 and 50).

  Awesome! Now let's try categorical plotting to examine the results in a more structured manner. First, plot the trace lengths.

  x = 'Trace lengths'
  
  g = sns.FacetGrid(data=corr_out,
                    col='Simulated $D$',
                    col_wrap=2,
                    sharex=True,
                    aspect=1.5,
                    height=5,
                    legend_out=True)
  g.map_dataframe(sns.boxplot,
        x=x,
        y='Traces used',
        hue='Simulated $D_{{clust}}$',
        palette='colorblind',
        showfliers=False)
  g.add_legend(title='Simulated $D_{{clust}}$')
  g.map_dataframe(sns.stripplot,
        x=x,
        y='Traces used',
        hue='Simulated $D_{{clust}}$',
        dodge=True,
        palette=sns.color_palette(['0.3']),
        size=4,
        jitter=0.2)
  g.set_xlabels(x)
  g.tight_layout()
  plt.show()
  

• Then, plot the diffusion rates and transit times. Since they follow a log normal distribution, use a log scale.

  # '$D$ in $\\frac{{\mu m^2}}{{s}}$'
  # '$\\tau_{{D}}$ in $ms$'
  x = '$\\tau_{{D}}$ in $ms$'
  
  g = sns.FacetGrid(data=corr_out,
                    col='Simulated $D$',
                    col_wrap=2,
                    sharex=False,  # False for x=D or x=tau, True for x=Trace lengths
                    aspect=1.5,
                    height=5,
                    legend_out=True)
  g.map_dataframe(sns.boxplot,
        x=x,
        y='Traces used',
        hue='Simulated $D_{{clust}}$',
        palette='colorblind',
        showfliers=False).set(xscale = 'log')
  g.add_legend(title='Simulated $D_{{clust}}$')
  g.map_dataframe(sns.stripplot,
        x=x,
        y='Traces used',
        hue='Simulated $D_{{clust}}$',
        dodge=True,
        palette=sns.color_palette(['0.3']),
        size=4,
        jitter=0.2).set(xscale = 'log')
  g.set_xlabels(x)
  g.tight_layout()
  plt.show()
  

  

• Now let's look at the corrections a little closer.

  corr_scatter = corr_out[corr_out['Traces used'].isin(['corrected by labels (control)', 'corrected by prediction'])]
  corr_scatter
  

  	Simulated \(D\)	Simulated \(D\_{{clust}}\)	nmol	\(D\) in \(\frac{{\mu m\^2}}{{s}}\)	\(\tau\_{{D}}\) in \(ms\)	Trace lengths	Traces used
  6000	0.069	1.0	679.0	0.051680	218.095043	14468	corrected by labels (control)
  6001	0.069	1.0	679.0	0.050863	221.596899	15380	corrected by labels (control)
  6002	0.069	1.0	679.0	0.060130	187.443486	14908	corrected by labels (control)
  6003	0.069	1.0	679.0	0.087759	128.431237	15230	corrected by labels (control)
  6004	0.069	1.0	679.0	0.144906	77.781595	14930	corrected by labels (control)
  …	…	…	…	…	…	…	…
  11995	0.400	0.1	1939.0	0.658104	17.126562	11187	corrected by prediction
  11996	0.400	0.1	1939.0	0.336804	33.464773	15720	corrected by prediction
  11997	0.400	0.1	1939.0	0.456482	24.691128	15292	corrected by prediction
  11998	0.400	0.1	1939.0	0.411872	27.365437	12516	corrected by prediction
  11999	0.400	0.1	1939.0	0.244884	46.026059	12986	corrected by prediction

  6000 rows × 7 columns

• This scatterplot shows Diffusion rates / transit times against trace lengths.

  # '$D$ in $\\frac{{\mu m^2}}{{s}}$'
  # '$\\tau_{{D}}$ in $ms$'
  x = '$\\tau_{{D}}$ in $ms$'
  
  g = sns.FacetGrid(data=corr_scatter,
                    row='Simulated $D$',
                    col='Traces used',
                    sharex='row',
                    aspect=1,
                    height=4,
                    legend_out=True,
                    margin_titles=True)
  g.map_dataframe(sns.scatterplot,
        x=x,
        y='Trace lengths',
        hue='Simulated $D_{{clust}}$',
        palette='colorblind').set(xscale = 'log')
  g.add_legend(title='Simulated $D_{{clust}}$')
  g.set_xlabels(x)
  g.tight_layout()
  plt.show()
  

• I concatenated the resulting 2 plots into 1 (left diffusion rates, right transit times). Here it can be nicely seen, that they are just two representations of the same dynamic phenomenon.